[GitHub] metron issue #801: METRON-1254: Conditionals as map keys do not function in ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/801 +1 by inspection ---
[GitHub] metron pull request #801: METRON-1254: Conditionals as map keys do not funct...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/801#discussion_r147259905 --- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java --- @@ -218,6 +218,25 @@ public void testVariablesUsed() { } } + @Test + public void testConditionalsAsMapKeys() { +{ + String query = "{ ( RET_TRUE() && y < 50 ) : 'info', y >= 50 : 'warn'}"; + Mapret = (Map)run(query, ImmutableMap.of("y", 50)); + Assert.assertEquals(ret.size(), 2); + Assert.assertEquals("warn", ret.get(true)); + Assert.assertEquals("info", ret.get(false)); +} + } + + + @Test + public void testConditionalsAsFunctionArgs() { +{ + String query = "RET_TRUE(y < 10)"; + Assert.assertTrue((boolean)run(query, ImmutableMap.of("y", 50))); +} + } --- End diff -- I was thinking of something else, but on second thought I don't think it is valid. ---
[GitHub] metron pull request #801: METRON-1254: Conditionals as map keys do not funct...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/801#discussion_r147257992 --- Diff: metron-stellar/stellar-common/src/main/antlr4/org/apache/metron/stellar/common/generated/Stellar.g4 --- @@ -249,6 +253,7 @@ identifier_operand : | NULL #NullConst | EXISTS LPAREN IDENTIFIER RPAREN #ExistsFunc | LPAREN conditional_expr RPAREN #condExpr_paren --- End diff -- Oh man, sorry, I did know that from the match work. ---
[GitHub] metron pull request #801: METRON-1254: Conditionals as map keys do not funct...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/801#discussion_r147257285 --- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java --- @@ -218,6 +218,25 @@ public void testVariablesUsed() { } } + @Test + public void testConditionalsAsMapKeys() { +{ + String query = "{ ( RET_TRUE() && y < 50 ) : 'info', y >= 50 : 'warn'}"; + Mapret = (Map)run(query, ImmutableMap.of("y", 50)); + Assert.assertEquals(ret.size(), 2); + Assert.assertEquals("warn", ret.get(true)); + Assert.assertEquals("info", ret.get(false)); +} + } + + + @Test + public void testConditionalsAsFunctionArgs() { +{ + String query = "RET_TRUE(y < 10)"; + Assert.assertTrue((boolean)run(query, ImmutableMap.of("y", 50))); +} + } --- End diff -- Well, we know that lambdas can be arguments for functions (because of `MAP` and `REDUCE`) ---
[GitHub] metron pull request #801: METRON-1254: Conditionals as map keys do not funct...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/801#discussion_r147257082 --- Diff: metron-stellar/stellar-common/src/main/antlr4/org/apache/metron/stellar/common/generated/Stellar.g4 --- @@ -249,6 +253,7 @@ identifier_operand : | NULL #NullConst | EXISTS LPAREN IDENTIFIER RPAREN #ExistsFunc | LPAREN conditional_expr RPAREN #condExpr_paren --- End diff -- If you specify one named function, you have to name them all. What would you like to name it? I didn't spend too much time on it since I knew I wasn't implementing it, admittedly. ---
[GitHub] metron pull request #818: METRON-1284: Remove extraneous dead query in Elast...
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/818 METRON-1284: Remove extraneous dead query in ElasticsearchDao ## Contributor Comments Delete a pointless query. Given that it's essentially just a noop (we query ES and then do nothing with the result other than retrieve a field), there's not really anything to add tests about and it didn't break existing tests. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron remove_dead_code Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/818.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #818 commit 5001a6d6ea282a06831be1f25e7ed21dc8ddef0e Author: justinjleetDate: 2017-10-26T19:21:23Z Remove extraneous ES call ---
[GitHub] metron issue #812: METRON-1273: Website documentation link should point to t...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/812 +1 Ran up, Docs Home [Learn More] takes you to https://metron.apache.org/current-book/index.html ---
[GitHub] metron pull request #801: METRON-1254: Conditionals as map keys do not funct...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/801#discussion_r147238667 --- Diff: metron-stellar/stellar-common/src/main/antlr4/org/apache/metron/stellar/common/generated/Stellar.g4 --- @@ -249,6 +253,7 @@ identifier_operand : | NULL #NullConst | EXISTS LPAREN IDENTIFIER RPAREN #ExistsFunc | LPAREN conditional_expr RPAREN #condExpr_paren --- End diff -- Why are we specifying the #func function? * We don't implement it * The name should be better ---
[GitHub] metron pull request #801: METRON-1254: Conditionals as map keys do not funct...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/801#discussion_r147239180 --- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java --- @@ -218,6 +218,25 @@ public void testVariablesUsed() { } } + @Test + public void testConditionalsAsMapKeys() { +{ + String query = "{ ( RET_TRUE() && y < 50 ) : 'info', y >= 50 : 'warn'}"; + Mapret = (Map)run(query, ImmutableMap.of("y", 50)); + Assert.assertEquals(ret.size(), 2); + Assert.assertEquals("warn", ret.get(true)); + Assert.assertEquals("info", ret.get(false)); +} + } + + + @Test + public void testConditionalsAsFunctionArgs() { +{ + String query = "RET_TRUE(y < 10)"; + Assert.assertTrue((boolean)run(query, ImmutableMap.of("y", 50))); +} + } --- End diff -- Do we need lambda with args tests for this as well? ---
[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/817 Hi @anandsubbu - Just a few issues that came to mind. These may or may not be problems, so I just wanted to open them for discussion. 1. This will reinstall the same templates each time indexing is started. This may unexpectedly overwrite changes made by the user to their index templates. Many of the other install actions use a file as an indicator so that it only performs the action once. Maybe you could do that too? 2. If Elasticsearch is not running, then the Indexing topology will fail to start. While normally this is probably OK, it might not be a dependency that we want to introduce. What if someone has configured their topology to not write to ES and is using an alternative indexing destination? What if you made it so that the topology could still start, even if Elasticsearch is not available? Maybe a "best effort" to install the templates, but if it fails, move on. 3. ---
[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/817 The templates are installed from ansible right now for the vagrant builds. Wouldn't we need to remove that step? Have you tested full_dev ? ---
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user iraghumitra commented on the issue: https://github.com/apache/metron/pull/803 I had to force push the last merge since I was facing issues with github. ---
[GitHub] metron pull request #817: METRON-1283: Install Elasticsearch template as a p...
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/817 METRON-1283: Install Elasticsearch template as a part of the mpack startup scripts ## Contributor Comments For a Metron multi-node deployment using mpack, the Elasticsearch template is required to be installed manually post-setup. These templates are required for the proper working of, for e.g. the Alerts UI. In the event that these templates are not installed, and if data is ingested, these would not be shown in the Alerts UI, since there would be missing fields without the template files (E.g. snort alert indices are not displayed in the Alerts UI, since it is missing the "alerts" field from the mapping). In such a case, one needs to install the templates, delete all indices for the given parser and re-ingest data again into the parser for it to appear in the Alerts UI. Further, the indices from all the parsers will have to be deleted and re-ingested again which could be a tedious job in the event that this step was missed out by chance. I have also seen other ill-effects from having stale indices for parsers that was created before template install. While documenting the template installation is a good practice, nothing would more failsafe than installing the template as a part of the mpack startup scripts itself. I have added the ES template install step just before starting the Indexing topology, since I found this to be the closest related step when this could be done. Do let me know if this needs to be moved to a different point in the startup, and I can make the change. **Testing Done** * Built mpack with the changes * Deployed a 12-node Metron setup using the mpack * Validated that the start Indexing topology step also installs the Elasticsearch template files. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/anandsubbu/incubator-metron METRON-1283 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/817.patch To close this pull request, make a commit to your master/trunk branch with (at least) the
[GitHub] metron pull request #811: METRON-1272: Hide child alerts from searches and g...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/811 ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 +1 from me as well. Great job @justinleet ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/811 +1 This was more challenging than meets the eye. Thanks for working through this @justinleet . We have some use cases to figure out, but this is a good first step toward metaalerts. ---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/803#discussion_r147152861 --- Diff: metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts --- @@ -337,12 +343,67 @@ export class TreeViewComponent extends TableViewComponent implements OnChanges { }); } + canCreateMetaAlert(count: number) { +if (count > 999) { --- End diff -- @james-sirota You'd know a bit more about the practical usage pattern than I would. Do you have any input on this limit? I'm guessing we could/should probably drop it a bit to give ourselves a bit more safety factor (e.g. 200?), assuming that fits a practical pattern. ---
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
GitHub user iraghumitra reopened a pull request: https://github.com/apache/metron/pull/803 Metron-1252: Build ui for grouping alerts into meta alerts ## Contributor Comments The purpose of the PR is to provide GUI for grouping multiple alerts into a meta alert, the rest api for this is already available via [METRON-1158](https://issues.apache.org/jira/browse/METRON-1158) The current implementation has following features - Meta alert can be created from the tree view of alerts - Meta alerts can be viewed in the table view - The meta alert has a link icon on the left to denote that it is a meta alert - The meta alert has a expand/collapse icon to see all the alerts within it - The meta alert has a cumulative score (more on how this score is calculated is available in the original ticket) - Support to add/remove an alert to the meta alert from the table view - Change the state of meta alert - Add comments to a meta alert - Option to name a meta alert (This is just a convenience to refer to alerts) **Limitations** - Meta alerts cannot be viewed in the tree view - Adding comments/Status change is restricted to just meta alerts. You cannot perform these actions on alerts contained in meta alert - A meta alert can contain only 999 alerts - Delete of the entire meta alert is not supported yet **Next** - It will be nice to have a notification when entities are changed in the UI I noticed that search on GUID was not working before I fixed it in this PR. ~E2E tests are incoming I wanted to check if the community has any suggestion on this.~ ~PS: I had to comment one of the test cases since sort by guid is broke. I will raise a ticket for it~ ![image](https://user-images.githubusercontent.com/15019012/31682771-b9deb6da-b398-11e7-8681-6696fdcd1b6c.png) ![image](https://user-images.githubusercontent.com/15019012/31682796-cc5236f2-b398-11e7-8db0-7233b35ba7b8.png) ![image](https://user-images.githubusercontent.com/15019012/31682824-dda3c02e-b398-11e7-9369-119587041262.png) ![image](https://user-images.githubusercontent.com/15019012/31682846-ee2451fc-b398-11e7-998e-c4f441048ffe.png) ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your
[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...
Github user iraghumitra closed the pull request at: https://github.com/apache/metron/pull/803 ---
[GitHub] metron pull request #796: METRON-1224: Add time range selection to search co...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/796 ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 +1 ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 + 1. Gret job. all pass login to application â should display error message for invalid credentials â should login for valid credentials â should logout metron-alerts App â should have all the UI elements â should have all pagination controls and they should be working â should have all settings controls and they should be working â play pause should start polling and stop polling â should select columns from table configuration â should have all time-range controls â should have all time range values populated - 1 â should have all time range values populated - 2 â should have all time range values populated - 3 â should have all time range values populated - 4 â should disable date picker when timestamp is present in search â should have now included when to date is empty â should have all time-range included while searching metron-alerts configure table â should select columns from table configuration â should rename columns from table configuration metron-alerts Search â should display all the default values for saved searches â should have all save search controls and they save search should be working â should populate search items when selected on table â should delete search items from search box â should delete first search items from search box having multiple search fields â manually entering search queries to search box and pressing enter key should search metron-alerts tree view â should have all group by elements â drag and drop should change group order â should have group details for single group by â should have group details for multiple group by â should have sort working for group details for multiple sub groups â should have search working for group details for multiple sub groups metron-alerts facets â should display facets data â should expand all facets â should have all facet values â should collapse all facets metron-alerts alert status â should change alert status for multiple alerts to OPEN â should change alert status for multiple alerts to DISMISS â should change alert status for multiple alerts to ESCALATE â should change alert status for multiple alerts to RESOLVE â should change alert status for multiple alerts to OPEN in tree view metron-alerts alert status â should change alert statuses â should add comments for table view â should add comments for tree view Executed 42 of 42 specs SUCCESS in 5 mins 2 secs. [01:01:00] I/launcher - 0 instance(s) of WebDriver still running [01:01:00] I/launcher - chrome #01 passed ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 login to application â should display error message for invalid credentials â should login for valid credentials â should logout metron-alerts App â should have all the UI elements â should have all pagination controls and they should be working â should have all settings controls and they should be working â play pause should start polling and stop polling â should select columns from table configuration â sould have all time-range controls â sould have all time-range included while searching - Expected 'Alerts (0)' to equal 'Alerts (5)'. metron-alerts configure table â should select columns from table configuration metron-alerts Search â should display all the default values for saved searches â should have all save search controls and they save search should be working â should populate search items when selected on table â should delete search items from search box â should delete first search items from search box having multiple search fields â manually entering search queries to search box and pressing enter key should search metron-alerts tree view â should have all group by elements â drag and drop should change group order â should have group details for single group by â should have group details for multiple group by â should have sort working for group details for multiple sub groups â should have search working for group details for multiple sub groups metron-alerts facets â should display facets data â should expand all facets â should have all facet values â should collapse all facets metron-alerts alert status â should change alert status for multiple alerts to OPEN â should change alert status for multiple alerts to DISMISS â should change alert status for multiple alerts to ESCALATE â should change alert status for multiple alerts to RESOLVE â should change alert status for multiple alerts to OPEN in tree view - Failed: element not visible (Session info: chrome=61.0.3163.100) (Driver info: chromedriver=2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.12.6 x86_64) metron-alerts alert status â should change alert statuses â should add comments for table view â should add comments for tree view ** *Failures* ** 1) metron-alerts App sould have all time-range included while searching - Expected 'Alerts (0)' to equal 'Alerts (5)'. 2) metron-alerts alert status should change alert status for multiple alerts to OPEN in tree view - Failed: element not visible (Session info: chrome=61.0.3163.100) (Driver info: chromedriver=2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.12.6 x86_64) ---