[jira] [Created] (METRON-1802) When switching parser topologies, some of the original parser topologies can fail to be shut down properly
Anand Subramanian created METRON-1802: - Summary: When switching parser topologies, some of the original parser topologies can fail to be shut down properly Key: METRON-1802 URL: https://issues.apache.org/jira/browse/METRON-1802 Project: Metron Issue Type: Bug Reporter: Anand Subramanian This defect was reported by [~nickwallen] when testing PR[ #1215|[https://github.com/apache/metron/pull/1215].] Steps to reproduce: # On full-dev, start with default parsers; [bro,snort] # Change Metron Parsers setting to say ["bro"] and save configuration # Restart the Parser topology to apply the config change. # Observe that the original "snort" topology is never shutdown. One would expect this to be shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1802) When switching parser topologies, some of the original parser topologies can fail to be shut down properly
[ https://issues.apache.org/jira/browse/METRON-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1802: -- Description: This defect was reported by [~nickwallen] when testing PR[ #1215|[https://github.com/apache/metron/pull/1215].] Steps to reproduce: # On full-dev, start with default parsers; [bro,snort] # Change Metron Parsers setting to say ["bro"] under Services -> Metron -> Configs -> 'Parsers' tab -> 'Metron Parsers' field. Save configuration and restart required services to apply the config change. # Observe that the original "snort" topology is never shutdown. One would expect this to be shutdown. was: This defect was reported by [~nickwallen] when testing PR[ #1215|[https://github.com/apache/metron/pull/1215].] Steps to reproduce: # On full-dev, start with default parsers; [bro,snort] # Change Metron Parsers setting to say ["bro"] and save configuration # Restart the Parser topology to apply the config change. # Observe that the original "snort" topology is never shutdown. One would expect this to be shutdown. > When switching parser topologies, some of the original parser topologies can > fail to be shut down properly > -- > > Key: METRON-1802 > URL: https://issues.apache.org/jira/browse/METRON-1802 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Priority: Major > > This defect was reported by [~nickwallen] when testing PR[ > #1215|[https://github.com/apache/metron/pull/1215].] > Steps to reproduce: > # On full-dev, start with default parsers; [bro,snort] > # Change Metron Parsers setting to say ["bro"] under Services -> Metron -> > Configs -> 'Parsers' tab -> 'Metron Parsers' field. Save configuration and > restart required services to apply the config change. > # Observe that the original "snort" topology is never shutdown. One would > expect this to be shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Hi @nickwallen , thank you for the review. > @nickwallen One thing I notice right off is that we did not add any documentation for the parsers fields in Ambari. Would it make sense to add a brief description to the pop-up text describing how a user can define parsers? Sure, makes sense. I have also added a section to the Parser Chaining README as well with examples. Please have a look. > @nickwallen: When switching parser topologies, some of the original parser topologies can fail to be shut down properly. > I have found that the same issue occurs in master. Since that is the case, we could choose to tackle this as a separate ticket. It is your choice. I created [METRON-1802](https://issues.apache.org/jira/browse/METRON-1802) so this can be fixed outside of this PR. ---
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636668#comment-16636668 ] ASF GitHub Bot commented on METRON-1798: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Hi @nickwallen , thank you for the review. > @nickwallen One thing I notice right off is that we did not add any documentation for the parsers fields in Ambari. Would it make sense to add a brief description to the pop-up text describing how a user can define parsers? Sure, makes sense. I have also added a section to the Parser Chaining README as well with examples. Please have a look. > @nickwallen: When switching parser topologies, some of the original parser topologies can fail to be shut down properly. > I have found that the same issue occurs in master. Since that is the case, we could choose to tackle this as a separate ticket. It is your choice. I created [METRON-1802](https://issues.apache.org/jira/browse/METRON-1802) so this can be fixed outside of this PR. > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...
Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/1215 ---
[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...
GitHub user anandsubbu reopened a pull request: https://github.com/apache/metron/pull/1215 METRON-1798: Add mpack support for parser aggregation ## Contributor Comments This pull request allows users to submit an aggregated parser topology. ## Testing Steps 1. Spin up full-dev. 2. Stop the "Metron Parsers" service so that existing parser topologies are killed/stopped. 3. Go to Ambari -> Metron -> Configs -> Parsers 4. Change the "Metron Parsers" value as: "bro,snort", "yaf (For example) 5. Save changes and restart the Metron Parsers service. 6. Go to Storm UI, and verify that the aggregated topologies viz. "bro__snort" and "yaf" are started. ## Testing Done ### 1. Full Dev * Set the Metron Parsers value as "bro,snort.yaf" and restarted the service. * A single aggregated topology is seen to be started: ![image](https://user-images.githubusercontent.com/20395490/46216724-dbb13a00-c35d-11e8-9651-b39d0766208f.png) * Verified stop 'Metron Parsers' service to check that the aggregated topology is stopped properly * Verified restart 'Metron Parsers' service to check that the aggregated topology is restarted properly ### 2. Multi-node setup * Set the Metron Parsers value as: "bro,snort,yaf","bro,yaf","snort,yaf",yaf,snort * The appropriate aggregated and single topologies are started: ![image](https://user-images.githubusercontent.com/20395490/46217047-84f83000-c35e-11e8-87e3-994b72d06cf3.png) * Changed the parser list as: bro,snort,yaf * Restarted the parsers service to validate that the three parser topologies are started individually. ![image](https://user-images.githubusercontent.com/20395490/46223825-0dcc9700-c372-11e8-9027-ad6ad1577f5f.png) ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1798 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1215
[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Re-run travis ---
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636692#comment-16636692 ] ASF GitHub Bot commented on METRON-1798: Github user anandsubbu closed the pull request at: https://github.com/apache/metron/pull/1215 > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636694#comment-16636694 ] ASF GitHub Bot commented on METRON-1798: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Re-run travis > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636695#comment-16636695 ] ASF GitHub Bot commented on METRON-1798: GitHub user anandsubbu reopened a pull request: https://github.com/apache/metron/pull/1215 METRON-1798: Add mpack support for parser aggregation ## Contributor Comments This pull request allows users to submit an aggregated parser topology. ## Testing Steps 1. Spin up full-dev. 2. Stop the "Metron Parsers" service so that existing parser topologies are killed/stopped. 3. Go to Ambari -> Metron -> Configs -> Parsers 4. Change the "Metron Parsers" value as: "bro,snort", "yaf (For example) 5. Save changes and restart the Metron Parsers service. 6. Go to Storm UI, and verify that the aggregated topologies viz. "bro__snort" and "yaf" are started. ## Testing Done ### 1. Full Dev * Set the Metron Parsers value as "bro,snort.yaf" and restarted the service. * A single aggregated topology is seen to be started: ![image](https://user-images.githubusercontent.com/20395490/46216724-dbb13a00-c35d-11e8-9651-b39d0766208f.png) * Verified stop 'Metron Parsers' service to check that the aggregated topology is stopped properly * Verified restart 'Metron Parsers' service to check that the aggregated topology is restarted properly ### 2. Multi-node setup * Set the Metron Parsers value as: "bro,snort,yaf","bro,yaf","snort,yaf",yaf,snort * The appropriate aggregated and single topologies are started: ![image](https://user-images.githubusercontent.com/20395490/46217047-84f83000-c35e-11e8-87e3-994b72d06cf3.png) * Changed the parser list as: bro,snort,yaf * Restarted the parsers service to validate that the three parser topologies are started individually. ![image](https://user-images.githubusercontent.com/20395490/46223825-0dcc9700-c372-11e8-9027-ad6ad1577f5f.png) ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1798 Alternatively you ca
[jira] [Created] (METRON-1803) Integrate Cypress tests with Travis
Tibor Meller created METRON-1803: Summary: Integrate Cypress tests with Travis Key: METRON-1803 URL: https://issues.apache.org/jira/browse/METRON-1803 Project: Metron Issue Type: Improvement Reporter: Tibor Meller Assignee: Tibor Meller -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1215 +1 This is really good fix @anandsubbu . Thanks for the easily grokable docs and clean code. ---
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636819#comment-16636819 ] ASF GitHub Bot commented on METRON-1798: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1215 +1 This is really good fix @anandsubbu . Thanks for the easily grokable docs and clean code. > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1215: METRON-1798: Add mpack support for parser aggregation
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Thanks again for the review, @nickwallen ! ---
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636945#comment-16636945 ] ASF GitHub Bot commented on METRON-1798: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1215 Thanks again for the review, @nickwallen ! > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1219: METRON-1796: [UI] Migrate off moment.js
GitHub user ruffle1986 opened a pull request: https://github.com/apache/metron/pull/1219 METRON-1796: [UI] Migrate off moment.js ## Contributor Comments [DISCUSS] thread on the dev mailing list: https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E Original ASF Jira ticket: https://issues.apache.org/jira/browse/METRON-1796 The purpose of this PR is the complete removal of [moment.js](http://momentjs.com/) from the code base and replacing it with either native functions or [date-fns](https://date-fns.org/), another date manipulation library. **Motivation** In the Metron Alerts UI, we use a few functions only from moment.js to deal with time like formatting or displaying the relative time from "now". In order to do that, we have to import the entire library which causes a significant footprint in the compiled production bundle. Sometimes they can be replaced with native built-in solutions, sometimes they cannot but we can use date-fns which has a smaller size comparing to moment.js As of date-fns v2 is in alpha phase, I rather chose the latest stable version which is 1.29.0. Tree-shaking is currently not supported in version 1 but [it will be in version 2](https://date-fns.org/v2.0.0-alpha.9/docs/ECMAScript-Modules). So once it's released and we can upgrade to it, we can see more size loss in the bundle size. Angular uses Webpack under the hood so if you're interested in what tree-shaking is and how Webpack does it, [follow this link](https://webpack.js.org/guides/tree-shaking/) for further details. **Changes included** - Every code used moment.js is replaced with a built-in function or the appropriate function from date-fns. I highly recommend you to go through all the commits I've made one by one. They're straightforward and easier to understand which parts of the app are affected and in what way. **Testing the bundle file size** 1. Checkout `master` 1. Go to `metron-interface/metron-alerts` 1. Make sure you have the latest packages in the `node_modules` folder via `npm ci` 1. Run `npm run build` 1. Take a look at the size of the `main.js` file 1. Checkout `ruffle1986/METRON-1796` 1. Run `npm ci` (this will install the date-fns library and remove moment.js) 1. Run `npm run build` 1. Take a look at the size of the `main.js` file and compare it to the number you've got in step 4 Here's the output of `npm run build` on the `master` branch: ![screen shot 2018-10-03 at 13 22 29](https://user-images.githubusercontent.com/2196208/46411822-08c66980-c71d-11e8-964b-6884c69b9d28.png) And here's the output on this branch: ![screen shot 2018-10-03 at 13 28 08](https://user-images.githubusercontent.com/2196208/46411889-2d224600-c71d-11e8-9ad9-fb0fbd58807a.png) As you can see, by removing moment.js, we could reduce the bundle size to 1.65 MB which is **15.6%** comparing to the bundle with moment.js. Not so significant but still. You've probably noticed the warning message on the second picture below the output of the build process. This is the result of compiling `pikaday-time` into our bundle via Angular. For the record, it's totally fine. Moment.js is just an optional dependency of [Pikaday](https://dbushell.com/Pikaday/), it works perfectly fine without it. This warning message is there because Pikaday [checks whether it's a Node.js environment and since it is, it wants to load moment from the node_modules folder](https://github.com/dbushell/Pikaday/blob/master/pikaday.js#L12-L16). However Pikaday doesn't throw and fails silently, the Angular compiler is clever enough to catch this and kindly warn you about it. But it doesn't cause any negative effect in the Metron UI because (again) moment.js is not a required dependency of Pikaday. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen
[jira] [Commented] (METRON-1796) [UI] Migrate off moment.js
[ https://issues.apache.org/jira/browse/METRON-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636946#comment-16636946 ] ASF GitHub Bot commented on METRON-1796: GitHub user ruffle1986 opened a pull request: https://github.com/apache/metron/pull/1219 METRON-1796: [UI] Migrate off moment.js ## Contributor Comments [DISCUSS] thread on the dev mailing list: https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E Original ASF Jira ticket: https://issues.apache.org/jira/browse/METRON-1796 The purpose of this PR is the complete removal of [moment.js](http://momentjs.com/) from the code base and replacing it with either native functions or [date-fns](https://date-fns.org/), another date manipulation library. **Motivation** In the Metron Alerts UI, we use a few functions only from moment.js to deal with time like formatting or displaying the relative time from "now". In order to do that, we have to import the entire library which causes a significant footprint in the compiled production bundle. Sometimes they can be replaced with native built-in solutions, sometimes they cannot but we can use date-fns which has a smaller size comparing to moment.js As of date-fns v2 is in alpha phase, I rather chose the latest stable version which is 1.29.0. Tree-shaking is currently not supported in version 1 but [it will be in version 2](https://date-fns.org/v2.0.0-alpha.9/docs/ECMAScript-Modules). So once it's released and we can upgrade to it, we can see more size loss in the bundle size. Angular uses Webpack under the hood so if you're interested in what tree-shaking is and how Webpack does it, [follow this link](https://webpack.js.org/guides/tree-shaking/) for further details. **Changes included** - Every code used moment.js is replaced with a built-in function or the appropriate function from date-fns. I highly recommend you to go through all the commits I've made one by one. They're straightforward and easier to understand which parts of the app are affected and in what way. **Testing the bundle file size** 1. Checkout `master` 1. Go to `metron-interface/metron-alerts` 1. Make sure you have the latest packages in the `node_modules` folder via `npm ci` 1. Run `npm run build` 1. Take a look at the size of the `main.js` file 1. Checkout `ruffle1986/METRON-1796` 1. Run `npm ci` (this will install the date-fns library and remove moment.js) 1. Run `npm run build` 1. Take a look at the size of the `main.js` file and compare it to the number you've got in step 4 Here's the output of `npm run build` on the `master` branch: ![screen shot 2018-10-03 at 13 22 29](https://user-images.githubusercontent.com/2196208/46411822-08c66980-c71d-11e8-964b-6884c69b9d28.png) And here's the output on this branch: ![screen shot 2018-10-03 at 13 28 08](https://user-images.githubusercontent.com/2196208/46411889-2d224600-c71d-11e8-9ad9-fb0fbd58807a.png) As you can see, by removing moment.js, we could reduce the bundle size to 1.65 MB which is **15.6%** comparing to the bundle with moment.js. Not so significant but still. You've probably noticed the warning message on the second picture below the output of the build process. This is the result of compiling `pikaday-time` into our bundle via Angular. For the record, it's totally fine. Moment.js is just an optional dependency of [Pikaday](https://dbushell.com/Pikaday/), it works perfectly fine without it. This warning message is there because Pikaday [checks whether it's a Node.js environment and since it is, it wants to load moment from the node_modules folder](https://github.com/dbushell/Pikaday/blob/master/pikaday.js#L12-L16). However Pikaday doesn't throw and fails silently, the Angular compiler is clever enough to catch this and kindly warn you about it. But it doesn't cause any negative effect in the Metron UI because (again) moment.js is not a required dependency of Pikaday. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](ht
[GitHub] metron pull request #1215: METRON-1798: Add mpack support for parser aggrega...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1215 ---
[jira] [Commented] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636948#comment-16636948 ] ASF GitHub Bot commented on METRON-1798: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1215 > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1749) Update Angular to latest release in Management UI
[ https://issues.apache.org/jira/browse/METRON-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636951#comment-16636951 ] ASF GitHub Bot commented on METRON-1749: Github user sardell commented on the issue: https://github.com/apache/metron/pull/1217 @mmiklavc Thanks for starting to take a look at this PR! You're correct that all these changes are related to the upgrade. This means removing deprecated items, updating syntax, utilizing updated or new features where we should and accommodating for much stricter type checking and compiling. > Update Angular to latest release in Management UI > - > > Key: METRON-1749 > URL: https://issues.apache.org/jira/browse/METRON-1749 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > > Currently, the Management UI is on Angular v2. Not only has there been many > updates and improvements to Angular, that release is no longer supported by > the Angular team. This means critical fixes and security patches are no > longer being done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1217: METRON-1749: Update Angular to latest release in Managem...
Github user sardell commented on the issue: https://github.com/apache/metron/pull/1217 @mmiklavc Thanks for starting to take a look at this PR! You're correct that all these changes are related to the upgrade. This means removing deprecated items, updating syntax, utilizing updated or new features where we should and accommodating for much stricter type checking and compiling. ---
[GitHub] metron pull request #1219: METRON-1796: [UI] Migrate off moment.js
Github user ruffle1986 commented on a diff in the pull request: https://github.com/apache/metron/pull/1219#discussion_r222308215 --- Diff: metron-interface/metron-alerts/package.json --- @@ -22,17 +22,17 @@ "@angular/platform-browser": "^6.1.6", "@angular/platform-browser-dynamic": "^6.1.6", "@angular/router": "^6.1.6", +"@ruffle1986/pikaday-time": "^1.6.1", "@types/bootstrap": "^4.1.1", "@types/jquery": "^3.3.4", "ace-builds": "^1.2.6", "ajv": "^6.5.1", "angular-confirmation-popover": "^4.2.0", "bootstrap": "4.0.0-alpha.6", "core-js": "^2.4.1", +"date-fns": "^1.29.0", "font-awesome": "^4.7.0", -"moment": "^2.22.2", "ng2-dragula": "^1.5.0", -"pikaday-time": "^1.6.1", --- End diff -- pikaday-time is an extension of [Pikaday](https://github.com/dbushell/Pikaday). Pikaday is a great library with zero dependency. Pikaday-time is extends its functionality with time selection. However moment is just an optional dependency of Pikaday, there's an unpleasant side-effect in Pikaday-time where moment.js is set as a dependency in the package.json. So everytime we install pikaday-time, moment.js will be installed as well. [I've tried to reach out the maintainer of the pikaday-time](https://github.com/owenmead/Pikaday/issues/60) library but he's not so active on Github and the library is no longer maintained anyway so it makes things a bit complicated. Hopefully we'll manage this and a patch for this will be introduced shortly. Until then, [I've forked the pikaday-time repo](https://github.com/owenmead/Pikaday/compare/master...ruffle1986:master), made the changes and [published it on npm under my name](https://www.npmjs.com/package/@ruffle1986/pikaday-time). Originally I wanted to published it under [hortonworks](https://www.npmjs.com/org/hortonworks) but I don't know who have access to give me publish rights. The only change I made is setting moment.js as a `peerDependency` insteadOf setting it as an `optionalDependency`. Don't let the name `optionalDependency` mislead you. [It's basically a dependency but if it's cannot be found on npm on any given registries, npm install doesn't fail](https://docs.npmjs.com/files/package.json#optionaldependencies). As soon as this issue is fixed and published on npm, we can remove the scoped pikaday-time and switch back to the original package. ---
[GitHub] metron pull request #1219: METRON-1796: [UI] Migrate off moment.js
Github user ruffle1986 commented on a diff in the pull request: https://github.com/apache/metron/pull/1219#discussion_r222315533 --- Diff: metron-interface/metron-alerts/package.json --- @@ -46,9 +46,7 @@ "@types/ace": "0.0.32", "@types/es6-promise": "0.0.33", "@types/jasmine": "2.5.38", -"@types/moment": "^2.13.0", "@types/node": "~6.0.60", -"@types/pikaday-time": "^1.4.2", --- End diff -- The reason why I removed this is because moment.js is the dependency of this to make it able to add the type `moment` to the return value of the `getMoment` method. Therefore moment.js has to be inside the node_modules folder but we wan't to avoid it to be there. I don't see any benefits of having the type definitions of pikaday-time in the code base. ---
[GitHub] metron pull request #1219: METRON-1796: [UI] Migrate off moment.js
Github user ruffle1986 commented on a diff in the pull request: https://github.com/apache/metron/pull/1219#discussion_r222317156 --- Diff: metron-interface/metron-alerts/src/app/utils/utils.spec.ts --- @@ -0,0 +1,215 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import {Utils} from './utils' + +describe('timeRangeToDisplayStr', () => { --- End diff -- [timeRangeToDisplayStr](https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/src/app/utils/utils.ts#L77-L206) relies heavily on moment.js so before touching the implementation I covered it with unit tests to make sure that the util produces the exact same result after the removal of moment.js. ---
[jira] [Commented] (METRON-1796) [UI] Migrate off moment.js
[ https://issues.apache.org/jira/browse/METRON-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636991#comment-16636991 ] ASF GitHub Bot commented on METRON-1796: Github user ruffle1986 commented on a diff in the pull request: https://github.com/apache/metron/pull/1219#discussion_r222317156 --- Diff: metron-interface/metron-alerts/src/app/utils/utils.spec.ts --- @@ -0,0 +1,215 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import {Utils} from './utils' + +describe('timeRangeToDisplayStr', () => { --- End diff -- [timeRangeToDisplayStr](https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/src/app/utils/utils.ts#L77-L206) relies heavily on moment.js so before touching the implementation I covered it with unit tests to make sure that the util produces the exact same result after the removal of moment.js. > [UI] Migrate off moment.js > -- > > Key: METRON-1796 > URL: https://issues.apache.org/jira/browse/METRON-1796 > Project: Metron > Issue Type: Improvement >Reporter: Tamas Fodor >Assignee: Tamas Fodor >Priority: Minor > > Remove Moment.js and replace with another smaller library. > Moment.js requires us to import the entire library vs. a few necessary > modules. > Moment.js can prevent bundlers from supporting tree-shaking. > By removing Moment.js, we can decrease our overall bundle size and prevent > issues with tree-shaking in the future. > Here you can find the discussion on the mailing list: > https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1796) [UI] Migrate off moment.js
[ https://issues.apache.org/jira/browse/METRON-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636992#comment-16636992 ] ASF GitHub Bot commented on METRON-1796: Github user ruffle1986 commented on a diff in the pull request: https://github.com/apache/metron/pull/1219#discussion_r222315533 --- Diff: metron-interface/metron-alerts/package.json --- @@ -46,9 +46,7 @@ "@types/ace": "0.0.32", "@types/es6-promise": "0.0.33", "@types/jasmine": "2.5.38", -"@types/moment": "^2.13.0", "@types/node": "~6.0.60", -"@types/pikaday-time": "^1.4.2", --- End diff -- The reason why I removed this is because moment.js is the dependency of this to make it able to add the type `moment` to the return value of the `getMoment` method. Therefore moment.js has to be inside the node_modules folder but we wan't to avoid it to be there. I don't see any benefits of having the type definitions of pikaday-time in the code base. > [UI] Migrate off moment.js > -- > > Key: METRON-1796 > URL: https://issues.apache.org/jira/browse/METRON-1796 > Project: Metron > Issue Type: Improvement >Reporter: Tamas Fodor >Assignee: Tamas Fodor >Priority: Minor > > Remove Moment.js and replace with another smaller library. > Moment.js requires us to import the entire library vs. a few necessary > modules. > Moment.js can prevent bundlers from supporting tree-shaking. > By removing Moment.js, we can decrease our overall bundle size and prevent > issues with tree-shaking in the future. > Here you can find the discussion on the mailing list: > https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1796) [UI] Migrate off moment.js
[ https://issues.apache.org/jira/browse/METRON-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636993#comment-16636993 ] ASF GitHub Bot commented on METRON-1796: Github user ruffle1986 commented on a diff in the pull request: https://github.com/apache/metron/pull/1219#discussion_r222308215 --- Diff: metron-interface/metron-alerts/package.json --- @@ -22,17 +22,17 @@ "@angular/platform-browser": "^6.1.6", "@angular/platform-browser-dynamic": "^6.1.6", "@angular/router": "^6.1.6", +"@ruffle1986/pikaday-time": "^1.6.1", "@types/bootstrap": "^4.1.1", "@types/jquery": "^3.3.4", "ace-builds": "^1.2.6", "ajv": "^6.5.1", "angular-confirmation-popover": "^4.2.0", "bootstrap": "4.0.0-alpha.6", "core-js": "^2.4.1", +"date-fns": "^1.29.0", "font-awesome": "^4.7.0", -"moment": "^2.22.2", "ng2-dragula": "^1.5.0", -"pikaday-time": "^1.6.1", --- End diff -- pikaday-time is an extension of [Pikaday](https://github.com/dbushell/Pikaday). Pikaday is a great library with zero dependency. Pikaday-time is extends its functionality with time selection. However moment is just an optional dependency of Pikaday, there's an unpleasant side-effect in Pikaday-time where moment.js is set as a dependency in the package.json. So everytime we install pikaday-time, moment.js will be installed as well. [I've tried to reach out the maintainer of the pikaday-time](https://github.com/owenmead/Pikaday/issues/60) library but he's not so active on Github and the library is no longer maintained anyway so it makes things a bit complicated. Hopefully we'll manage this and a patch for this will be introduced shortly. Until then, [I've forked the pikaday-time repo](https://github.com/owenmead/Pikaday/compare/master...ruffle1986:master), made the changes and [published it on npm under my name](https://www.npmjs.com/package/@ruffle1986/pikaday-time). Originally I wanted to published it under [hortonworks](https://www.npmjs.com/org/hortonworks) but I don't know who have access to give me publish rights. The only change I made is setting moment.js as a `peerDependency` insteadOf setting it as an `optionalDependency`. Don't let the name `optionalDependency` mislead you. [It's basically a dependency but if it's cannot be found on npm on any given registries, npm install doesn't fail](https://docs.npmjs.com/files/package.json#optionaldependencies). As soon as this issue is fixed and published on npm, we can remove the scoped pikaday-time and switch back to the original package. > [UI] Migrate off moment.js > -- > > Key: METRON-1796 > URL: https://issues.apache.org/jira/browse/METRON-1796 > Project: Metron > Issue Type: Improvement >Reporter: Tamas Fodor >Assignee: Tamas Fodor >Priority: Minor > > Remove Moment.js and replace with another smaller library. > Moment.js requires us to import the entire library vs. a few necessary > modules. > Moment.js can prevent bundlers from supporting tree-shaking. > By removing Moment.js, we can decrease our overall bundle size and prevent > issues with tree-shaking in the future. > Here you can find the discussion on the mailing list: > https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1217: METRON-1749: Update Angular to latest release in Managem...
Github user sardell commented on the issue: https://github.com/apache/metron/pull/1217 @mmiklavc I was unable to find a discuss thread. I understand this is standard for a change like this and I would be more than happy to start one if the community desires, otherwise I'm hoping we can have that discussion here. There is now a brief explanation in the Contributor Comments for why we should upgrade the Management UI like we did the Alerts UI. ---
[jira] [Commented] (METRON-1749) Update Angular to latest release in Management UI
[ https://issues.apache.org/jira/browse/METRON-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637036#comment-16637036 ] ASF GitHub Bot commented on METRON-1749: Github user sardell commented on the issue: https://github.com/apache/metron/pull/1217 @mmiklavc I was unable to find a discuss thread. I understand this is standard for a change like this and I would be more than happy to start one if the community desires, otherwise I'm hoping we can have that discussion here. There is now a brief explanation in the Contributor Comments for why we should upgrade the Management UI like we did the Alerts UI. > Update Angular to latest release in Management UI > - > > Key: METRON-1749 > URL: https://issues.apache.org/jira/browse/METRON-1749 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > > Currently, the Management UI is on Angular v2. Not only has there been many > updates and improvements to Angular, that release is no longer supported by > the Angular team. This means critical fixes and security patches are no > longer being done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @mmiklavc I looked through the validation stuff more, I think that validation is the way to go here. The grok parser will add invalid message for each exception, parser failure, and then in the validation call fail those messages. It will have to be done so that the returned message makes sense when it is sent to the error topic. What do you think? @cestella ? ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[ https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637040#comment-16637040 ] ASF GitHub Bot commented on METRON-1761: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @mmiklavc I looked through the validation stuff more, I think that validation is the way to go here. The grok parser will add invalid message for each exception, parser failure, and then in the validation call fail those messages. It will have to be done so that the returned message makes sense when it is sent to the error topic. What do you think? @cestella ? > Allow a grok statement to be applied to each line in a file. > > > Key: METRON-1761 > URL: https://issues.apache.org/jira/browse/METRON-1761 > Project: Metron > Issue Type: Improvement >Reporter: Laurens Vets >Assignee: Otto Fowler >Priority: Minor > > Make grok work where each line in incoming logs is a separate unit to be > parsed. > This would for instance allow NiFi to pick up log files (whereby each line is > to be parsed separately) and send them to Metron without having to split the > content. > Example content of a log file where a grok statement needs to be applied to > each line: > {code:java} > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/ > HTTP/1.1" "curl/7.38.0" - - > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ > HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2 > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - - > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.001065 0.15 0.23 - - 57 502 "- - - " "-" > ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1213 @merrimanr @nickwallen @justinleet - I just want to be sure that I'm following this correctly. This particular callback appears to be a synchronous, blocking callback. The crux of the implementation is that you're passing in a function pointer to be executed when the parsing either succeeds or fails in order to decouple the parsing logic from Storm itself (e.g. tuples, etc.)? ---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637041#comment-16637041 ] ASF GitHub Bot commented on METRON-1681: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1213 @merrimanr @nickwallen @justinleet - I just want to be sure that I'm following this correctly. This particular callback appears to be a synchronous, blocking callback. The crux of the implementation is that you're passing in a function pointer to be executed when the parsing either succeeds or fails in order to decouple the parsing logic from Storm itself (e.g. tuples, etc.)? > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 That sounds correct to me. Specifically decoupling the Kafka writing step since our writing implementation requires a tuple. ---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637047#comment-16637047 ] ASF GitHub Bot commented on METRON-1681: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 That sounds correct to me. Specifically decoupling the Kafka writing step since our writing implementation requires a tuple. > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1804) Update version to 0.6.1
Justin Leet created METRON-1804: --- Summary: Update version to 0.6.1 Key: METRON-1804 URL: https://issues.apache.org/jira/browse/METRON-1804 Project: Metron Issue Type: Task Reporter: Justin Leet Assignee: Justin Leet Post release version bump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1220: METRON-1804: Update version to 0.6.1
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1220 METRON-1804: Update version to 0.6.1 ## Contributor Comments Updating version to 0.6.1 post release. Changed per https://cwiki.apache.org/confluence/display/METRON/Change+the+Build+Version+Number 0.6.0 references should be gone (barring site stuff, other dependencies like Zeppelin, etc.). Pom and doc references should all be updated. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron version061 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1220.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1220 commit efb46065aa4a1b5618489c7e7948859f60c2f6f6 Author: justinjleet Date: 2018-09-18T16:03:58Z Upping version ot 0.6.1 ---
[jira] [Commented] (METRON-1804) Update version to 0.6.1
[ https://issues.apache.org/jira/browse/METRON-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637120#comment-16637120 ] ASF GitHub Bot commented on METRON-1804: GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1220 METRON-1804: Update version to 0.6.1 ## Contributor Comments Updating version to 0.6.1 post release. Changed per https://cwiki.apache.org/confluence/display/METRON/Change+the+Build+Version+Number 0.6.0 references should be gone (barring site stuff, other dependencies like Zeppelin, etc.). Pom and doc references should all be updated. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron version061 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1220.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1220 commit efb46065aa4a1b5618489c7e7948859f60c2f6f6 Author: justinjleet Date: 2018-09-18T16:03:58Z Upping version ot 0.6.1 > Update version to 0.6.1 > --- > > Key: METRON-1804 > URL: https://issues.apache.org/jira/browse/METRON-1804 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Post release version bump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1220: METRON-1804: Update version to 0.6.1
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1220 Looks good. Thanks! +1 ---
[jira] [Commented] (METRON-1804) Update version to 0.6.1
[ https://issues.apache.org/jira/browse/METRON-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637306#comment-16637306 ] ASF GitHub Bot commented on METRON-1804: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1220 Looks good. Thanks! +1 > Update version to 0.6.1 > --- > > Key: METRON-1804 > URL: https://issues.apache.org/jira/browse/METRON-1804 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Post release version bump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/1213#discussion_r222397888 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java --- @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.parsers; + +import com.github.benmanes.caffeine.cache.Cache; +import org.apache.commons.lang3.StringUtils; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.FieldValidator; +import org.apache.metron.common.configuration.ParserConfigurations; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.message.metadata.RawMessage; +import org.apache.metron.common.utils.ReflectionUtils; +import org.apache.metron.parsers.filters.Filters; +import org.apache.metron.parsers.interfaces.MessageFilter; +import org.apache.metron.parsers.interfaces.MessageParser; +import org.apache.metron.parsers.topology.ParserComponent; +import org.apache.metron.stellar.common.CachingStellarProcessor; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.json.simple.JSONObject; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.function.Consumer; +import java.util.function.Supplier; +import java.util.stream.Collectors; + +public class ParserRunner implements Serializable { --- End diff -- Maybe this should be "Context" ---
[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/1213#discussion_r222376135 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java --- @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.parsers; + +import com.github.benmanes.caffeine.cache.Cache; +import org.apache.commons.lang3.StringUtils; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.FieldValidator; +import org.apache.metron.common.configuration.ParserConfigurations; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.message.metadata.RawMessage; +import org.apache.metron.common.utils.ReflectionUtils; +import org.apache.metron.parsers.filters.Filters; +import org.apache.metron.parsers.interfaces.MessageFilter; +import org.apache.metron.parsers.interfaces.MessageParser; +import org.apache.metron.parsers.topology.ParserComponent; +import org.apache.metron.stellar.common.CachingStellarProcessor; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.json.simple.JSONObject; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.function.Consumer; +import java.util.function.Supplier; +import java.util.stream.Collectors; + +public class ParserRunner implements Serializable { + + protected transient Consumer onSuccess; + protected transient Consumer onError; + + private HashSet sensorTypes; + private Map sensorToParserComponentMap; + + // Stellar variables + private transient Context stellarContext; --- End diff -- Why the change to making this transient? ---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637312#comment-16637312 ] ASF GitHub Bot commented on METRON-1681: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/1213#discussion_r222376135 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java --- @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.parsers; + +import com.github.benmanes.caffeine.cache.Cache; +import org.apache.commons.lang3.StringUtils; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.FieldValidator; +import org.apache.metron.common.configuration.ParserConfigurations; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.message.metadata.RawMessage; +import org.apache.metron.common.utils.ReflectionUtils; +import org.apache.metron.parsers.filters.Filters; +import org.apache.metron.parsers.interfaces.MessageFilter; +import org.apache.metron.parsers.interfaces.MessageParser; +import org.apache.metron.parsers.topology.ParserComponent; +import org.apache.metron.stellar.common.CachingStellarProcessor; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.json.simple.JSONObject; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.function.Consumer; +import java.util.function.Supplier; +import java.util.stream.Collectors; + +public class ParserRunner implements Serializable { + + protected transient Consumer onSuccess; + protected transient Consumer onError; + + private HashSet sensorTypes; + private Map sensorToParserComponentMap; + + // Stellar variables + private transient Context stellarContext; --- End diff -- Why the change to making this transient? > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637313#comment-16637313 ] ASF GitHub Bot commented on METRON-1681: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/1213#discussion_r222397888 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java --- @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.parsers; + +import com.github.benmanes.caffeine.cache.Cache; +import org.apache.commons.lang3.StringUtils; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.FieldValidator; +import org.apache.metron.common.configuration.ParserConfigurations; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.message.metadata.RawMessage; +import org.apache.metron.common.utils.ReflectionUtils; +import org.apache.metron.parsers.filters.Filters; +import org.apache.metron.parsers.interfaces.MessageFilter; +import org.apache.metron.parsers.interfaces.MessageParser; +import org.apache.metron.parsers.topology.ParserComponent; +import org.apache.metron.stellar.common.CachingStellarProcessor; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.json.simple.JSONObject; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.function.Consumer; +import java.util.function.Supplier; +import java.util.stream.Collectors; + +public class ParserRunner implements Serializable { --- End diff -- Maybe this should be "Context" > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1184 @ottobackwards I see what you're saying. It looks like that could definitely work. Thinking out loud here, but might that conflate the semantics of our validation a bit? Validate currently does things like ensure that a timestamp exists on the message, though I don't see why we couldn't expand it to validations outside of our global Metron context. One class that might be worth checking out is the unified enrichment topology. This was changed to include a parallel enricher that handles errors and message results in an EnrichmentResult class. 1. https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java#L270 2. https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java#L63 It looks to me like there might be some possible collaboration opportunity and overlap with the work you're doing here and the work @merrimanr is doing on this PR - https://github.com/apache/metron/pull/1213#pullrequestreview-161248142 I'm just wondering if we might be able to kill 2 birds with one stone. We probably don't want to change the MessageParser interface, but maybe we can manage the bulk processing through a more generalized bridge between the ParserBolt and parser implementations. I haven't dug too deep into implementation feasibility, but it seems worth considering. ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[ https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637336#comment-16637336 ] ASF GitHub Bot commented on METRON-1761: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1184 @ottobackwards I see what you're saying. It looks like that could definitely work. Thinking out loud here, but might that conflate the semantics of our validation a bit? Validate currently does things like ensure that a timestamp exists on the message, though I don't see why we couldn't expand it to validations outside of our global Metron context. One class that might be worth checking out is the unified enrichment topology. This was changed to include a parallel enricher that handles errors and message results in an EnrichmentResult class. 1. https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java#L270 2. https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java#L63 It looks to me like there might be some possible collaboration opportunity and overlap with the work you're doing here and the work @merrimanr is doing on this PR - https://github.com/apache/metron/pull/1213#pullrequestreview-161248142 I'm just wondering if we might be able to kill 2 birds with one stone. We probably don't want to change the MessageParser interface, but maybe we can manage the bulk processing through a more generalized bridge between the ParserBolt and parser implementations. I haven't dug too deep into implementation feasibility, but it seems worth considering. > Allow a grok statement to be applied to each line in a file. > > > Key: METRON-1761 > URL: https://issues.apache.org/jira/browse/METRON-1761 > Project: Metron > Issue Type: Improvement >Reporter: Laurens Vets >Assignee: Otto Fowler >Priority: Minor > > Make grok work where each line in incoming logs is a separate unit to be > parsed. > This would for instance allow NiFi to pick up log files (whereby each line is > to be parsed separately) and send them to Metron without having to split the > content. > Example content of a log file where a grok statement needs to be applied to > each line: > {code:java} > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/ > HTTP/1.1" "curl/7.38.0" - - > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ > HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2 > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - - > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.001065 0.15 0.23 - - 57 502 "- - - " "-" > ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 @MohanDV , as a part of my commit, I have moved all the UI related elements from REST to the PCAP config tab. In my understanding, the following parameters are not directly related to the PCAP core service, but are functionally related to REST (used by the PCAP query panel UI): * pdml.script.path * base.path * base.interim.result.path * final.output.path * page.size * yarn.queue * finalizer.threadpool.size For this reason, I did not move these properties out of the rest_application.yml. > @MohanDV Changing a REST Setting triggers Ambari to Restart PCAP Topology With my current changes, modifying any config setting in the PCAP tab will require restarting both Metron PCAP and Metron REST services--which is necessary since user can change a PCAP-service related property or a PCAP-REST related property (or both). The converse, which you noted, i.e. changing any REST config does not require restarting the PCAP topology. Here's a screenshot after making changes to REST config... ![image](https://user-images.githubusercontent.com/20395490/46430735-8e6f0700-c767-11e8-8308-728771635480.png) ---
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637343#comment-16637343 ] ASF GitHub Bot commented on METRON-1695: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 @MohanDV , as a part of my commit, I have moved all the UI related elements from REST to the PCAP config tab. In my understanding, the following parameters are not directly related to the PCAP core service, but are functionally related to REST (used by the PCAP query panel UI): * pdml.script.path * base.path * base.interim.result.path * final.output.path * page.size * yarn.queue * finalizer.threadpool.size For this reason, I did not move these properties out of the rest_application.yml. > @MohanDV Changing a REST Setting triggers Ambari to Restart PCAP Topology With my current changes, modifying any config setting in the PCAP tab will require restarting both Metron PCAP and Metron REST services--which is necessary since user can change a PCAP-service related property or a PCAP-REST related property (or both). The converse, which you noted, i.e. changing any REST config does not require restarting the PCAP topology. Here's a screenshot after making changes to REST config... ![image](https://user-images.githubusercontent.com/20395490/46430735-8e6f0700-c767-11e8-8308-728771635480.png) > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1190#discussion_r222420697 --- Diff: metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java --- @@ -282,4 +276,27 @@ public Document getLatest(final String guid, String sensorType) throws IOExcepti public List getIndices() { return indices; } + + private Document getDocument(List documentContainers) throws IOException { +Document ret = null; +List error = new ArrayList<>(); +for(DocumentContainer dc : documentContainers) { + if(dc.getException().isPresent()) { +Throwable e = dc.getException().get(); +error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e)); + } + else { +if(dc.getDocument().isPresent()) { + Document d = dc.getDocument().get(); + if(ret == null || ret.getTimestamp() < d.getTimestamp()) { +ret = d; + } +} + } +} --- End diff -- Hmm. Interesting. I don't really understand what happens in that else condition then. The OCD side of me wants to make sure I understand this to ensure there is not a pre-existing bug. But at the same time I can't fault you for this. All you did was refactor this code out into a method. It seems that a `DocumentContainer` should either have an exception (`dc.getException().isPresent()`) or it should have a document (`dc.getDocument.isPresent()`). We would hit the else when neither of those are the case. So under what conditions would a `DocumentContainer` have neither of those? I guess a `null` Document or null Throwable is added to the `DocumentContainer`? I hope this is what we expect to happen. ``` if(dc.getException().isPresent()) { .. } else if(dc.getDocument.isPresent()) { .. } else { // when does this occur? what do we expect to happen? } ``` ---
[GitHub] metron issue #1207: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 @nickwallen suggested a great idea of making use of parser aggregation and thus starting the default parsers on full-dev as an aggregated parser topology. The latest commit makes use of PR #1215 which provides that ability. Now, on full-dev we start a single aggregated topology for the default configured sensors viz. "bro,snort,yaf". ## Testing Done - Spun up full-dev and noticed that a single topology named `bro__snort__yaf` is started - Verified Alerts UI has data from all three sensors: ![image](https://user-images.githubusercontent.com/20395490/46431349-33d6aa80-c769-11e8-8b6e-33940ef9b9eb.png) ---
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637354#comment-16637354 ] ASF GitHub Bot commented on METRON-1695: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1207 @nickwallen suggested a great idea of making use of parser aggregation and thus starting the default parsers on full-dev as an aggregated parser topology. The latest commit makes use of PR #1215 which provides that ability. Now, on full-dev we start a single aggregated topology for the default configured sensors viz. "bro,snort,yaf". ## Testing Done - Spun up full-dev and noticed that a single topology named `bro__snort__yaf` is started - Verified Alerts UI has data from all three sensors: ![image](https://user-images.githubusercontent.com/20395490/46431349-33d6aa80-c769-11e8-8b6e-33940ef9b9eb.png) > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[ https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637352#comment-16637352 ] ASF GitHub Bot commented on METRON-1771: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1190#discussion_r222420697 --- Diff: metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java --- @@ -282,4 +276,27 @@ public Document getLatest(final String guid, String sensorType) throws IOExcepti public List getIndices() { return indices; } + + private Document getDocument(List documentContainers) throws IOException { +Document ret = null; +List error = new ArrayList<>(); +for(DocumentContainer dc : documentContainers) { + if(dc.getException().isPresent()) { +Throwable e = dc.getException().get(); +error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e)); + } + else { +if(dc.getDocument().isPresent()) { + Document d = dc.getDocument().get(); + if(ret == null || ret.getTimestamp() < d.getTimestamp()) { +ret = d; + } +} + } +} --- End diff -- Hmm. Interesting. I don't really understand what happens in that else condition then. The OCD side of me wants to make sure I understand this to ensure there is not a pre-existing bug. But at the same time I can't fault you for this. All you did was refactor this code out into a method. It seems that a `DocumentContainer` should either have an exception (`dc.getException().isPresent()`) or it should have a document (`dc.getDocument.isPresent()`). We would hit the else when neither of those are the case. So under what conditions would a `DocumentContainer` have neither of those? I guess a `null` Document or null Throwable is added to the `DocumentContainer`? I hope this is what we expect to happen. ``` if(dc.getException().isPresent()) { .. } else if(dc.getDocument.isPresent()) { .. } else { // when does this occur? what do we expect to happen? } ``` > Update REST endpoints to support eventually consistent UI updates > - > > Key: METRON-1771 > URL: https://issues.apache.org/jira/browse/METRON-1771 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Priority: Major > > Currently the REST endpoints that perform document updates either return > true/false or nothing. This puts the responsibility of retrieving the > updated state of the object on the client in a separate call or > optimistically applying the changes and reverting when an update fails. This > can be problematic if a client attempts to get the current state immediately > after an update and the change isn't visible yet in the back end. > Ideally they should return the updated state of the object, eliminating the > need to look up the updated state in a separate call. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1190#discussion_r222421545 --- Diff: metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.elasticsearch.dao; + +import org.apache.metron.indexing.dao.AccessConfig; +import org.apache.metron.indexing.dao.UpdateDaoTest; +import org.apache.metron.indexing.dao.update.UpdateDao; +import org.elasticsearch.client.transport.TransportClient; +import org.junit.Before; + +import static org.mockito.Mockito.mock; + +public class ElasticsearchUpdateDaoTest extends UpdateDaoTest { --- End diff -- I think you missed the javadoc here. ---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[ https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637356#comment-16637356 ] ASF GitHub Bot commented on METRON-1771: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1190#discussion_r222421545 --- Diff: metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.elasticsearch.dao; + +import org.apache.metron.indexing.dao.AccessConfig; +import org.apache.metron.indexing.dao.UpdateDaoTest; +import org.apache.metron.indexing.dao.update.UpdateDao; +import org.elasticsearch.client.transport.TransportClient; +import org.junit.Before; + +import static org.mockito.Mockito.mock; + +public class ElasticsearchUpdateDaoTest extends UpdateDaoTest { --- End diff -- I think you missed the javadoc here. > Update REST endpoints to support eventually consistent UI updates > - > > Key: METRON-1771 > URL: https://issues.apache.org/jira/browse/METRON-1771 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Priority: Major > > Currently the REST endpoints that perform document updates either return > true/false or nothing. This puts the responsibility of retrieving the > updated state of the object on the client in a separate call or > optimistically applying the changes and reverting when an update fails. This > can be problematic if a client attempts to get the current state immediately > after an update and the change isn't visible yet in the back end. > Ideally they should return the updated state of the object, eliminating the > need to look up the updated state in a separate call. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1213#discussion_r222423266 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java --- @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.parsers; + +import com.github.benmanes.caffeine.cache.Cache; +import org.apache.commons.lang3.StringUtils; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.FieldValidator; +import org.apache.metron.common.configuration.ParserConfigurations; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.message.metadata.RawMessage; +import org.apache.metron.common.utils.ReflectionUtils; +import org.apache.metron.parsers.filters.Filters; +import org.apache.metron.parsers.interfaces.MessageFilter; +import org.apache.metron.parsers.interfaces.MessageParser; +import org.apache.metron.parsers.topology.ParserComponent; +import org.apache.metron.stellar.common.CachingStellarProcessor; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.json.simple.JSONObject; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.function.Consumer; +import java.util.function.Supplier; +import java.util.stream.Collectors; + +public class ParserRunner implements Serializable { --- End diff -- I don't get `ParserContext`. The `ParserRunner` is something that runs/executes parsers, no? It doesn't provide context for something else to run parsers. But maybe I am misunderstanding why you think `ParserContext` makes sense. Can you explain your thoughts on that? ---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637363#comment-16637363 ] ASF GitHub Bot commented on METRON-1681: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1213#discussion_r222423266 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java --- @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.parsers; + +import com.github.benmanes.caffeine.cache.Cache; +import org.apache.commons.lang3.StringUtils; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.FieldValidator; +import org.apache.metron.common.configuration.ParserConfigurations; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.message.metadata.RawMessage; +import org.apache.metron.common.utils.ReflectionUtils; +import org.apache.metron.parsers.filters.Filters; +import org.apache.metron.parsers.interfaces.MessageFilter; +import org.apache.metron.parsers.interfaces.MessageParser; +import org.apache.metron.parsers.topology.ParserComponent; +import org.apache.metron.stellar.common.CachingStellarProcessor; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.json.simple.JSONObject; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.function.Consumer; +import java.util.function.Supplier; +import java.util.stream.Collectors; + +public class ParserRunner implements Serializable { --- End diff -- I don't get `ParserContext`. The `ParserRunner` is something that runs/executes parsers, no? It doesn't provide context for something else to run parsers. But maybe I am misunderstanding why you think `ParserContext` makes sense. Can you explain your thoughts on that? > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1213 > @mmiklavc Have you looked over this by chance [2]? The reason I bring this up is because it looks like it already addresses a number of the points/questions brought up in this PR's discussion. For example, there's a strategy for handling successful messages as well as error results [3]. If we're going through the effort of refactoring, it's probably worth our while to match the other idioms to some degree. It just makes continued maintenance that much easier going forward. Are you making the case for returning a `List` here then? Sounds like you're thinking along these lines? ``` ParserRunner { List execute(...); } ``` That would be similar to the unified enrichment work that you linked to in your comment. ``` ParallelEnricher { EnrichmentResult apply(...); } ``` ---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637376#comment-16637376 ] ASF GitHub Bot commented on METRON-1681: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1213 > @mmiklavc Have you looked over this by chance [2]? The reason I bring this up is because it looks like it already addresses a number of the points/questions brought up in this PR's discussion. For example, there's a strategy for handling successful messages as well as error results [3]. If we're going through the effort of refactoring, it's probably worth our while to match the other idioms to some degree. It just makes continued maintenance that much easier going forward. Are you making the case for returning a `List` here then? Sounds like you're thinking along these lines? ``` ParserRunner { List execute(...); } ``` That would be similar to the unified enrichment work that you linked to in your comment. ``` ParallelEnricher { EnrichmentResult apply(...); } ``` > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1184: METRON-1761, allow application of grok statement multipl...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @mmiklavc but we don't have messages to split, we have bytes. If we where going to leave the 'parser's as single object -> single result | single exceception', ie not change the interface and not subvert validate, then we would have to introduce 'splitStrategies' at the bolt. ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[ https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637393#comment-16637393 ] ASF GitHub Bot commented on METRON-1761: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1184 @mmiklavc but we don't have messages to split, we have bytes. If we where going to leave the 'parser's as single object -> single result | single exceception', ie not change the interface and not subvert validate, then we would have to introduce 'splitStrategies' at the bolt. > Allow a grok statement to be applied to each line in a file. > > > Key: METRON-1761 > URL: https://issues.apache.org/jira/browse/METRON-1761 > Project: Metron > Issue Type: Improvement >Reporter: Laurens Vets >Assignee: Otto Fowler >Priority: Minor > > Make grok work where each line in incoming logs is a separate unit to be > parsed. > This would for instance allow NiFi to pick up log files (whereby each line is > to be parsed separately) and send them to Metron without having to split the > content. > Example content of a log file where a grok statement needs to be applied to > each line: > {code:java} > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/ > HTTP/1.1" "curl/7.38.0" - - > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ > HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2 > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - - > 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 > 0.001065 0.15 0.23 - - 57 502 "- - - " "-" > ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1220: METRON-1804: Update version to 0.6.1
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1220 ---
[jira] [Commented] (METRON-1804) Update version to 0.6.1
[ https://issues.apache.org/jira/browse/METRON-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637415#comment-16637415 ] ASF GitHub Bot commented on METRON-1804: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1220 > Update version to 0.6.1 > --- > > Key: METRON-1804 > URL: https://issues.apache.org/jira/browse/METRON-1804 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Post release version bump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637431#comment-16637431 ] ASF GitHub Bot commented on METRON-1681: Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1213#discussion_r222440594 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java --- @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.parsers; + +import com.github.benmanes.caffeine.cache.Cache; +import org.apache.commons.lang3.StringUtils; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.FieldValidator; +import org.apache.metron.common.configuration.ParserConfigurations; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.message.metadata.RawMessage; +import org.apache.metron.common.utils.ReflectionUtils; +import org.apache.metron.parsers.filters.Filters; +import org.apache.metron.parsers.interfaces.MessageFilter; +import org.apache.metron.parsers.interfaces.MessageParser; +import org.apache.metron.parsers.topology.ParserComponent; +import org.apache.metron.stellar.common.CachingStellarProcessor; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.json.simple.JSONObject; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.function.Consumer; +import java.util.function.Supplier; +import java.util.stream.Collectors; + +public class ParserRunner implements Serializable { + + protected transient Consumer onSuccess; + protected transient Consumer onError; + + private HashSet sensorTypes; + private Map sensorToParserComponentMap; + + // Stellar variables + private transient Context stellarContext; --- End diff -- Context is not serializable. > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1213: METRON-1681: Decouple the ParserBolt from the Par...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1213#discussion_r222440594 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunner.java --- @@ -0,0 +1,234 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.parsers; + +import com.github.benmanes.caffeine.cache.Cache; +import org.apache.commons.lang3.StringUtils; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.FieldValidator; +import org.apache.metron.common.configuration.ParserConfigurations; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.message.metadata.RawMessage; +import org.apache.metron.common.utils.ReflectionUtils; +import org.apache.metron.parsers.filters.Filters; +import org.apache.metron.parsers.interfaces.MessageFilter; +import org.apache.metron.parsers.interfaces.MessageParser; +import org.apache.metron.parsers.topology.ParserComponent; +import org.apache.metron.stellar.common.CachingStellarProcessor; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.json.simple.JSONObject; + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.function.Consumer; +import java.util.function.Supplier; +import java.util.stream.Collectors; + +public class ParserRunner implements Serializable { + + protected transient Consumer onSuccess; + protected transient Consumer onError; + + private HashSet sensorTypes; + private Map sensorToParserComponentMap; + + // Stellar variables + private transient Context stellarContext; --- End diff -- Context is not serializable. ---
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 Thanks for pointing me to that @mmiklavc, I had not looked at it yet. I am in agreement that we should do our best to match other classes and patterns we already have. I will continue studying that to see where we can make things more consistent. If you can provide some examples that demonstrate your ideas that would help me understand it better. From what I can tell, the strategy pattern was used in the UnifiedEnrichmentBolt because there are 2 separate strategies for enriching a message: enrichment and threat intel. Do we need the strategy pattern here since there is only one parser running strategy? We could implement this as a strategy pattern in anticipation of one day needing another parser running strategy. Do you think it's worth doing now or later when we need it? The other difference I see is that Future objects are used to join the different messages after they are processed in parallel. I believe this is done because all enrichments/threat intel must be done before the message can be pieced back together and sent along. In this case we don't have that limitation. The documents that are parsed do not depend on each other and can be passed along as soon as they are processed. We are set up for post-processing these documents in parallel but I see that as a low-latency operation and I'm not sure it's worth the extra overhead. ---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637474#comment-16637474 ] ASF GitHub Bot commented on METRON-1681: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 Thanks for pointing me to that @mmiklavc, I had not looked at it yet. I am in agreement that we should do our best to match other classes and patterns we already have. I will continue studying that to see where we can make things more consistent. If you can provide some examples that demonstrate your ideas that would help me understand it better. From what I can tell, the strategy pattern was used in the UnifiedEnrichmentBolt because there are 2 separate strategies for enriching a message: enrichment and threat intel. Do we need the strategy pattern here since there is only one parser running strategy? We could implement this as a strategy pattern in anticipation of one day needing another parser running strategy. Do you think it's worth doing now or later when we need it? The other difference I see is that Future objects are used to join the different messages after they are processed in parallel. I believe this is done because all enrichments/threat intel must be done before the message can be pieced back together and sent along. In this case we don't have that limitation. The documents that are parsed do not depend on each other and can be passed along as soon as they are processed. We are set up for post-processing these documents in parallel but I see that as a low-latency operation and I'm not sure it's worth the extra overhead. > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1190#discussion_r222470180 --- Diff: metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java --- @@ -282,4 +276,27 @@ public Document getLatest(final String guid, String sensorType) throws IOExcepti public List getIndices() { return indices; } + + private Document getDocument(List documentContainers) throws IOException { +Document ret = null; +List error = new ArrayList<>(); +for(DocumentContainer dc : documentContainers) { + if(dc.getException().isPresent()) { +Throwable e = dc.getException().get(); +error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e)); + } + else { +if(dc.getDocument().isPresent()) { + Document d = dc.getDocument().get(); + if(ret == null || ret.getTimestamp() < d.getTimestamp()) { +ret = d; + } +} + } +} --- End diff -- No problem, don't mind working it out here and now. The case that causes the test to fail (although I'm not sure that was the intention of the test) is where 2 DAOs are contained in a MultiIndexDao and the first DAO returns an invalid result (no document or exception) while the second returns a valid result. I don't know what would cause a document to be missing both a document and an exception. The question is what should happen when this unexpected condition does happen? Should it blow up and throw and exception or return a document if at least one DAO returns a valid one. ---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[ https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637507#comment-16637507 ] ASF GitHub Bot commented on METRON-1771: Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1190#discussion_r222470180 --- Diff: metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java --- @@ -282,4 +276,27 @@ public Document getLatest(final String guid, String sensorType) throws IOExcepti public List getIndices() { return indices; } + + private Document getDocument(List documentContainers) throws IOException { +Document ret = null; +List error = new ArrayList<>(); +for(DocumentContainer dc : documentContainers) { + if(dc.getException().isPresent()) { +Throwable e = dc.getException().get(); +error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e)); + } + else { +if(dc.getDocument().isPresent()) { + Document d = dc.getDocument().get(); + if(ret == null || ret.getTimestamp() < d.getTimestamp()) { +ret = d; + } +} + } +} --- End diff -- No problem, don't mind working it out here and now. The case that causes the test to fail (although I'm not sure that was the intention of the test) is where 2 DAOs are contained in a MultiIndexDao and the first DAO returns an invalid result (no document or exception) while the second returns a valid result. I don't know what would cause a document to be missing both a document and an exception. The question is what should happen when this unexpected condition does happen? Should it blow up and throw and exception or return a document if at least one DAO returns a valid one. > Update REST endpoints to support eventually consistent UI updates > - > > Key: METRON-1771 > URL: https://issues.apache.org/jira/browse/METRON-1771 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Priority: Major > > Currently the REST endpoints that perform document updates either return > true/false or nothing. This puts the responsibility of retrieving the > updated state of the object on the client in a separate call or > optimistically applying the changes and reverting when an update fails. This > can be problematic if a client attempts to get the current state immediately > after an update and the change isn't visible yet in the back end. > Ideally they should return the updated state of the object, eliminating the > need to look up the updated state in a separate call. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1213: METRON-1681: Decouple the ParserBolt from the Parse exec...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 I just pushed a commit out based on some feedback from Nick: - The stellar context is now passed into the ParserRunner abstraction as a dependency. I imagine different environments will require different Stellar initialization strategies so this made sense to me. I also like it because it makes the abstraction simpler. - Changed ParserRunner to an interface with a single implementation. I could change this to ParserRunnerStrategy in case we think we might want to use that pattern in the future. Then all that would be missing is a ParserContext class (and possibly a ParserRunnerStrategies enum) which we could add now or when we need it. ---
[jira] [Commented] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637512#comment-16637512 ] ASF GitHub Bot commented on METRON-1681: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1213 I just pushed a commit out based on some feedback from Nick: - The stellar context is now passed into the ParserRunner abstraction as a dependency. I imagine different environments will require different Stellar initialization strategies so this made sense to me. I also like it because it makes the abstraction simpler. - Changed ParserRunner to an interface with a single implementation. I could change this to ParserRunnerStrategy in case we think we might want to use that pattern in the future. Then all that would be missing is a ParserContext class (and possibly a ParserRunnerStrategies enum) which we could add now or when we need it. > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Major > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1805) Provide a default value for the Storm topology.max.spout.pending setting
Ryan Merriman created METRON-1805: - Summary: Provide a default value for the Storm topology.max.spout.pending setting Key: METRON-1805 URL: https://issues.apache.org/jira/browse/METRON-1805 Project: Metron Issue Type: Improvement Reporter: Ryan Merriman Should we provide a default value for this property? Which topologies should that apply to? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1221: METRON-1805: Provide a default value for the Stor...
GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1221 METRON-1805: Provide a default value for the Storm topology.max.spout.pending setting ## Contributor Comments Users have reported problems in the random and batch indexing topologies when this setting is not set. The primary purpose of this PR is to start a discussion around providing a default value for the Storm topology.max.spout.pending setting in our topologies and implement the changes if we decide to do it. 1. Should we provide defaults or continue to defer to the Storm default where a maximum isn't enforced? 2. Which topologies should we provide defaults for? 3. What should the default values be for the topologies we include? ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1805 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1221.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1221 commit 6988e0607afc1cd402063e0696a1220cc134d9d1 Author: merrimanr Date: 2018-10-03T21:45:32Z initial commit ---
[jira] [Commented] (METRON-1805) Provide a default value for the Storm topology.max.spout.pending setting
[ https://issues.apache.org/jira/browse/METRON-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637544#comment-16637544 ] ASF GitHub Bot commented on METRON-1805: GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1221 METRON-1805: Provide a default value for the Storm topology.max.spout.pending setting ## Contributor Comments Users have reported problems in the random and batch indexing topologies when this setting is not set. The primary purpose of this PR is to start a discussion around providing a default value for the Storm topology.max.spout.pending setting in our topologies and implement the changes if we decide to do it. 1. Should we provide defaults or continue to defer to the Storm default where a maximum isn't enforced? 2. Which topologies should we provide defaults for? 3. What should the default values be for the topologies we include? ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1805 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1221.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1221 commit 6988e0607afc1cd402063e0696a1220cc134d9d1 Author: merrimanr Date: 2018-10-03T21:45:32Z initial commit > Provide a default value for the Storm topology.max.spout.pending setting > > > Key: METRON-1805 > URL: https://issues.apache.org/jira/browse/METRON-1805 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Priority: Major > > Should we provide a default value for this property? Which topologies should > that apply to? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1222: Updated the Readme.md for Regular expressions parser.
Github user jagdeepsingh2 commented on the issue: https://github.com/apache/metron/pull/1222 This PR needs to be ignored. ---
[GitHub] metron pull request #1222: Updated the Readme.md for Regular expressions par...
GitHub user jagdeepsingh2 opened a pull request: https://github.com/apache/metron/pull/1222 Updated the Readme.md for Regular expressions parser. Configuration field explanation for regular expressions parser. ## Contributor Comments [Please place any comments here. A description of the problem/enhancement, how to reproduce the issue, your testing methodology, etc.] ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jagdeepsingh2/metron patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1222.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1222 commit f888fde0e9b0d46889749ce753aa0183531e20a2 Author: jagdeepsingh2 <43630622+jagdeepsingh2@...> Date: 2018-10-04T02:45:52Z Updated the Readme.md for Regular expressions parser. Configuration field explanation for regular expressions parser. ---
[GitHub] metron pull request #1222: Updated the Readme.md for Regular expressions par...
Github user jagdeepsingh2 closed the pull request at: https://github.com/apache/metron/pull/1222 ---