[GitHub] metron issue #1010: METRON-1548: Remove hardcoded source:type from Alerts UI
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1010 Thanks for the contribution, getting this cleaned up is really helpful! Could you update https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration with the new config property? I'd also like to see the property be something like `source.type.field` to be a bit more explanatory and consistent with the other non-Object type fields. ---
[jira] [Commented] (METRON-1548) Alerts UI: Remove hardcoded source:type and other fields that may vary
[ https://issues.apache.org/jira/browse/METRON-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474122#comment-16474122 ] ASF GitHub Bot commented on METRON-1548: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1010 Thanks for the contribution, getting this cleaned up is really helpful! Could you update https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration with the new config property? I'd also like to see the property be something like `source.type.field` to be a bit more explanatory and consistent with the other non-Object type fields. > Alerts UI: Remove hardcoded source:type and other fields that may vary > -- > > Key: METRON-1548 > URL: https://issues.apache.org/jira/browse/METRON-1548 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Priority: Major > > In Solr, we use source.type instead of source:type (which was originally due > to ES limitations, we'd prefer it to be common across both). However, it's > hardcoded in the alerts UI to be source:type, so nothing in the UI has a > source type and it breaks other things like metaalerts (which rely on source > type for being able to pull the data together). > Any other configs / queries that would cause similar problems should be > updated appropriately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #278: Metron 451 SerDeUtils - java.lang.ClassNotFoundException:...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/278 -1 Based on the [Metron Development Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines) on Inactive PR. Missing contributor feedback. I will open an Apache Infra request to close this PR. ---
[GitHub] metron pull request #278: Metron 451 SerDeUtils - java.lang.ClassNotFoundExc...
Github user Humbedooh closed the pull request at: https://github.com/apache/metron/pull/278 ---
[jira] [Commented] (METRON-451) SerDeUtils - java.lang.ClassNotFoundException: org.apache.storm.shade.org.joda.time.DateTime
[ https://issues.apache.org/jira/browse/METRON-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474143#comment-16474143 ] ASF GitHub Bot commented on METRON-451: --- Github user Humbedooh closed the pull request at: https://github.com/apache/metron/pull/278 > SerDeUtils - java.lang.ClassNotFoundException: > org.apache.storm.shade.org.joda.time.DateTime > > > Key: METRON-451 > URL: https://issues.apache.org/jira/browse/METRON-451 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > Ran into this issue attempting to write profile data to HBase. > {code} > Exception in thread "main" java.lang.IllegalStateException: Unable to > serialize 4790 because org/apache/storm/shade/org/joda/time/DateTime > at org.apache.metron.common.utils.SerDeUtils.toBytes(SerDeUtils.java:233) > at > org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder.columns(ValueOnlyColumnBuilder.java:52) > at > org.apache.metron.profiler.client.hbase.HBaseProfileWriter.write(HBaseProfileWriter.java:92) > at > org.apache.metron.profiler.client.hbase.HBaseProfileWriter.write(HBaseProfileWriter.java:80) > at > org.apache.metron.profiler.client.hbase.ProfilerWriterCLI.main(ProfilerWriterCLI.java:149) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) > Caused by: java.lang.NoClassDefFoundError: > org/apache/storm/shade/org/joda/time/DateTime > at > org.apache.metron.common.utils.SerDeUtils$1.initialValue(SerDeUtils.java:87) > at > org.apache.metron.common.utils.SerDeUtils$1.initialValue(SerDeUtils.java:63) > at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180) > at java.lang.ThreadLocal.get(ThreadLocal.java:170) > at org.apache.metron.common.utils.SerDeUtils.toBytes(SerDeUtils.java:226) > ... 9 more > Caused by: java.lang.ClassNotFoundException: > org.apache.storm.shade.org.joda.time.DateTime > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 14 more > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #338: METRON-295: Script parsing bolt
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/338 -1 There has been no response from contributor. This PR will be closed according to the [Metron Development Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines) on Inactive PRs. ---
[jira] [Commented] (METRON-295) Script parsing bolt
[ https://issues.apache.org/jira/browse/METRON-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474146#comment-16474146 ] ASF GitHub Bot commented on METRON-295: --- Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/338 -1 There has been no response from contributor. This PR will be closed according to the [Metron Development Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines) on Inactive PRs. > Script parsing bolt > > > Key: METRON-295 > URL: https://issues.apache.org/jira/browse/METRON-295 > Project: Metron > Issue Type: New Feature >Affects Versions: 0.2.2BETA >Reporter: James Sirota >Assignee: Karthik Narayanan >Priority: Minor > Labels: newbie, platform > > In addition to having a Grok parsing bolt we need a bolt that can execute a > script in order to parse a telemetry. This way you can still script the > parsing for telemetries for which Grok expressions are too complex, but still > don't have to define a java parser -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #670: METRON-1065 grok pattern for cisco asa parser expects sys...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/670 -1 There has been no response from the contributor. This PR will be closed per the [Metron Development Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines) on Inactive PRs ---
[jira] [Commented] (METRON-1065) Grok pattern for Cisco ASA Parser expects syslog_pri
[ https://issues.apache.org/jira/browse/METRON-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474148#comment-16474148 ] ASF GitHub Bot commented on METRON-1065: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/670 -1 There has been no response from the contributor. This PR will be closed per the [Metron Development Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines) on Inactive PRs > Grok pattern for Cisco ASA Parser expects syslog_pri > > > Key: METRON-1065 > URL: https://issues.apache.org/jira/browse/METRON-1065 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.1 >Reporter: Bas van de Lustgraaf >Priority: Minor > > The current grok pattern `CISCO_TAGGED_SYSLOG` expects to have a syslog > priority present at the start of each message. Unfortunately, this is not > always the case. > *Currently supported:* > {noformat} > <162>Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from > 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside > {noformat} > *Not supported by the current Grok pattern:* > {noformat} > Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from > 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside > {noformat} > My suggestion would be to edit the `CISCO_TAGGED_SYSLOG` pattern to make the > following part optional: > {noformat} > <%{POSINT:syslog_pri}> > {noformat} > And grep the severity from the `%ASA-4-106023` part. The part between the > hyphens, is the severity (source > http://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs-sev-level.html). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #670: METRON-1065 grok pattern for cisco asa parser expe...
Github user basvdl closed the pull request at: https://github.com/apache/metron/pull/670 ---
[jira] [Commented] (METRON-1065) Grok pattern for Cisco ASA Parser expects syslog_pri
[ https://issues.apache.org/jira/browse/METRON-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474151#comment-16474151 ] ASF GitHub Bot commented on METRON-1065: Github user basvdl closed the pull request at: https://github.com/apache/metron/pull/670 > Grok pattern for Cisco ASA Parser expects syslog_pri > > > Key: METRON-1065 > URL: https://issues.apache.org/jira/browse/METRON-1065 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.1 >Reporter: Bas van de Lustgraaf >Priority: Minor > > The current grok pattern `CISCO_TAGGED_SYSLOG` expects to have a syslog > priority present at the start of each message. Unfortunately, this is not > always the case. > *Currently supported:* > {noformat} > <162>Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from > 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside > {noformat} > *Not supported by the current Grok pattern:* > {noformat} > Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from > 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside > {noformat} > My suggestion would be to edit the `CISCO_TAGGED_SYSLOG` pattern to make the > following part optional: > {noformat} > <%{POSINT:syslog_pri}> > {noformat} > And grep the severity from the `%ASA-4-106023` part. The part between the > hyphens, is the severity (source > http://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs-sev-level.html). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #670: METRON-1065 grok pattern for cisco asa parser expe...
GitHub user basvdl reopened a pull request: https://github.com/apache/metron/pull/670 METRON-1065 grok pattern for cisco asa parser expects syslog pri ## Contributor Comments Cisco ASA logs does not always contain a priority code at the beginnen of the event. Therefore I've changed the grok pattern to make this optional. `<164>Aug 05 2016 01:01:34: %ASA-4-106023: Deny tcp src Inside:10.30.9.121/54580 dst Outside:192.168.135.51/42028 by access-group "Inside_access_in" [0x962df600, 0x0]` <164> should be optional. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated **unit tests** and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/basvdl/metron METRON-1065_Grok_pattern_for_Cisco_ASA_Parser_expects_syslog_pri Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/670.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #670 commit c691e3374ea9e9bdb4c291071adae045516e744b Author: Bas van de Lustgraaf Date: 2017-07-26T14:04:05Z Priority for Cisco ASA logging is optional commit a731004e53b948979bdbb6d01bdd274702eef0c7 Author: Bas van de Lustgraaf Date: 2017-07-26T14:10:14Z Fixed formatting commit 69584cdee5098bcc6449c72f5a69129e20c23cd8 Author: Bas van de Lustgraaf Date: 2017-07-26T14:18:04Z Fix the license header after using the intelliJ auto format ---
[jira] [Commented] (METRON-295) Script parsing bolt
[ https://issues.apache.org/jira/browse/METRON-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474168#comment-16474168 ] ASF GitHub Bot commented on METRON-295: --- Github user Humbedooh closed the pull request at: https://github.com/apache/metron/pull/338 > Script parsing bolt > > > Key: METRON-295 > URL: https://issues.apache.org/jira/browse/METRON-295 > Project: Metron > Issue Type: New Feature >Affects Versions: 0.2.2BETA >Reporter: James Sirota >Assignee: Karthik Narayanan >Priority: Minor > Labels: newbie, platform > > In addition to having a Grok parsing bolt we need a bolt that can execute a > script in order to parse a telemetry. This way you can still script the > parsing for telemetries for which Grok expressions are too complex, but still > don't have to define a java parser -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #338: METRON-295: Script parsing bolt
Github user Humbedooh closed the pull request at: https://github.com/apache/metron/pull/338 ---
[GitHub] metron pull request #670: METRON-1065 grok pattern for cisco asa parser expe...
Github user Humbedooh closed the pull request at: https://github.com/apache/metron/pull/670 ---
[jira] [Commented] (METRON-1065) Grok pattern for Cisco ASA Parser expects syslog_pri
[ https://issues.apache.org/jira/browse/METRON-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474169#comment-16474169 ] ASF GitHub Bot commented on METRON-1065: GitHub user basvdl reopened a pull request: https://github.com/apache/metron/pull/670 METRON-1065 grok pattern for cisco asa parser expects syslog pri ## Contributor Comments Cisco ASA logs does not always contain a priority code at the beginnen of the event. Therefore I've changed the grok pattern to make this optional. `<164>Aug 05 2016 01:01:34: %ASA-4-106023: Deny tcp src Inside:10.30.9.121/54580 dst Outside:192.168.135.51/42028 by access-group "Inside_access_in" [0x962df600, 0x0]` <164> should be optional. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated **unit tests** and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/basvdl/metron METRON-1065_Grok_pattern_for_Cisco_ASA_Parser_expects_syslog_pri Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/670.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #670 commit c691e3374ea9e9bdb4c291071adae045516e744b Author: Bas van de Lustgraaf Date: 2017-07-26T14:04:05Z Priority for Cisco ASA logging is optional commit a731004e53b948979bdbb6d01bdd274702eef0c7 Author: Bas van de Lustgraaf Date: 2017-07-26T14:10:14Z Fixed formatting commit 69584cdee5098bcc6449c72f5a69129e20c23cd8 Author: Bas van de Lustgraaf Date: 2017-07-26T14:18:04Z Fix the license header after using the intelliJ auto format > Grok pattern for Cisco ASA Parser expects syslog_pri > > > Key: METRON-1065 > URL: https://issues.apache.org/jira/browse/METRON-1065 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.1 >Reporter: Bas van de Lustgraaf >Priority: Minor > > The current grok pattern `CISCO_TAGGED_SYSLOG` expects to have a syslog > priority present at the start of
[jira] [Commented] (METRON-1065) Grok pattern for Cisco ASA Parser expects syslog_pri
[ https://issues.apache.org/jira/browse/METRON-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474170#comment-16474170 ] ASF GitHub Bot commented on METRON-1065: Github user Humbedooh closed the pull request at: https://github.com/apache/metron/pull/670 > Grok pattern for Cisco ASA Parser expects syslog_pri > > > Key: METRON-1065 > URL: https://issues.apache.org/jira/browse/METRON-1065 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.1 >Reporter: Bas van de Lustgraaf >Priority: Minor > > The current grok pattern `CISCO_TAGGED_SYSLOG` expects to have a syslog > priority present at the start of each message. Unfortunately, this is not > always the case. > *Currently supported:* > {noformat} > <162>Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from > 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside > {noformat} > *Not supported by the current Grok pattern:* > {noformat} > Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from > 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside > {noformat} > My suggestion would be to edit the `CISCO_TAGGED_SYSLOG` pattern to make the > following part optional: > {noformat} > <%{POSINT:syslog_pri}> > {noformat} > And grep the severity from the `%ASA-4-106023` part. The part between the > hyphens, is the severity (source > http://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs-sev-level.html). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1004: METRON-1489: Retrofit UI tests to run reliably
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1004#discussion_r187948446 --- Diff: metron-interface/metron-alerts/e2e/alerts-list/meta-alerts/meta-alert.e2e-spec.ts --- @@ -19,139 +19,142 @@ import { MetronAlertsPage } from '../alerts-list.po'; import {customMatchers} from '../../matchers/custom-matchers'; import {LoginPage} from '../../login/login.po'; -import {loadTestData, deleteTestData} from '../../utils/e2e_util'; +import {loadTestData, deleteTestData, createMetaAlertsIndex} from '../../utils/e2e_util'; import {TreeViewPage} from '../tree-view/tree-view.po'; import {MetronAlertDetailsPage} from '../../alert-details/alert-details.po'; import {MetaAlertPage} from './meta-alert.po'; import {AlertFacetsPage} from '../alert-filters/alert-filters.po'; -describe('meta-alerts workflow', function() { +describe('Test spec for meta alerts workflow', function() { let detailsPage: MetronAlertDetailsPage; let tablePage: MetronAlertsPage; let metaAlertPage: MetaAlertPage; let treePage: TreeViewPage; let loginPage: LoginPage; let alertFacetsPage: AlertFacetsPage; - beforeAll(() => { -loadTestData(); - + beforeAll(async function() : Promise { loginPage = new LoginPage(); -loginPage.login(); tablePage = new MetronAlertsPage(); treePage = new TreeViewPage(); tablePage = new MetronAlertsPage(); metaAlertPage = new MetaAlertPage(); detailsPage = new MetronAlertDetailsPage(); alertFacetsPage = new AlertFacetsPage(); + +await createMetaAlertsIndex(); +await loadTestData(); +await loginPage.login(); }); - afterAll(() => { -loginPage.logout(); -deleteTestData(); + afterAll(async function() : Promise { +await loginPage.logout(); +await deleteTestData(); }); beforeEach(() => { jasmine.addMatchers(customMatchers); }); - it('should have all the steps for meta alerts workflow', () => { + it('should have all the steps for meta alerts workflow', async function() : Promise { let comment1 = 'This is a sample comment'; let userNameAndTimestamp = '- admin - a few seconds ago'; -let confirmText = 'Do you wish to create a meta alert with 113 selected alerts?'; +let confirmText = 'Do you wish to create a meta alert with 13 selected alerts?'; let dashRowValues = { - 'firstDashRow': ['0', '192.168.138.158', 'ALERTS', '113'], - 'secondDashRow': ['0', '192.168.66.1', 'ALERTS', '56'] + 'firstDashRow': ['0', 'runlove.us', 'ALERTS', '13'] }; -tablePage.navigateTo(); +await tablePage.navigateTo(); +expect(await tablePage.getChangesAlertTableTitle('Alerts (0)')).toEqual('Alerts (169)'); /* Create Meta Alert */ -treePage.selectGroup('ip_src_addr'); - expect(treePage.getDashGroupValues('192.168.138.158')).toEqualBcoz(dashRowValues.firstDashRow, 'First Dashrow to be present'); - expect(treePage.getDashGroupValues('192.168.66.1')).toEqualBcoz(dashRowValues.secondDashRow, 'Second Dashrow to be present'); +await treePage.selectGroup('host'); +expect(await treePage.getDashGroupValues('runlove.us')).toEqualBcoz(dashRowValues.firstDashRow, 'First Dashrow to be present'); -treePage.clickOnMergeAlerts('192.168.138.158'); -expect(treePage.getConfirmationText()).toEqualBcoz(confirmText, 'confirmation text to be present'); -treePage.clickNoForConfirmation(); +await treePage.clickOnMergeAlerts('runlove.us'); +expect(await treePage.getConfirmationText()).toEqualBcoz(confirmText, 'confirmation text to be present'); +await treePage.clickNoForConfirmation(); -treePage.clickOnMergeAlerts('192.168.138.158'); -treePage.clickYesForConfirmation(); +await treePage.clickOnMergeAlerts('runlove.us'); +await treePage.clickYesForConfirmation(); -treePage.waitForElementToDisappear('192.168.138.158'); +await treePage.waitForElementToDisappear('runlove.us'); -treePage.unGroup(); +await treePage.unGroup(); /* Table should have all alerts */ -tablePage.waitForMetaAlert(); -expect(tablePage.getPaginationText()).toEqualBcoz('1 - 25 of 57', 'pagination text to be present'); -expect(tablePage.getCellValue(0, 2, '(114)')).toContain('(113)', 'number of alerts in a meta alert should be correct'); -expect(tablePage.getNonHiddenRowCount()).toEqualBcoz(25, '25 rows to be visible'); -expect(tablePage.getAllRowsCount()).toEqualBcoz(138, '138 rows to be available'); -expect(tablePage.getHiddenRowCount()).toEqualBcoz(113, '1
[jira] [Commented] (METRON-1489) Retrofit UI tests to run reliably during nightly QE runs
[ https://issues.apache.org/jira/browse/METRON-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474183#comment-16474183 ] ASF GitHub Bot commented on METRON-1489: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1004#discussion_r187948446 --- Diff: metron-interface/metron-alerts/e2e/alerts-list/meta-alerts/meta-alert.e2e-spec.ts --- @@ -19,139 +19,142 @@ import { MetronAlertsPage } from '../alerts-list.po'; import {customMatchers} from '../../matchers/custom-matchers'; import {LoginPage} from '../../login/login.po'; -import {loadTestData, deleteTestData} from '../../utils/e2e_util'; +import {loadTestData, deleteTestData, createMetaAlertsIndex} from '../../utils/e2e_util'; import {TreeViewPage} from '../tree-view/tree-view.po'; import {MetronAlertDetailsPage} from '../../alert-details/alert-details.po'; import {MetaAlertPage} from './meta-alert.po'; import {AlertFacetsPage} from '../alert-filters/alert-filters.po'; -describe('meta-alerts workflow', function() { +describe('Test spec for meta alerts workflow', function() { let detailsPage: MetronAlertDetailsPage; let tablePage: MetronAlertsPage; let metaAlertPage: MetaAlertPage; let treePage: TreeViewPage; let loginPage: LoginPage; let alertFacetsPage: AlertFacetsPage; - beforeAll(() => { -loadTestData(); - + beforeAll(async function() : Promise { loginPage = new LoginPage(); -loginPage.login(); tablePage = new MetronAlertsPage(); treePage = new TreeViewPage(); tablePage = new MetronAlertsPage(); metaAlertPage = new MetaAlertPage(); detailsPage = new MetronAlertDetailsPage(); alertFacetsPage = new AlertFacetsPage(); + +await createMetaAlertsIndex(); +await loadTestData(); +await loginPage.login(); }); - afterAll(() => { -loginPage.logout(); -deleteTestData(); + afterAll(async function() : Promise { +await loginPage.logout(); +await deleteTestData(); }); beforeEach(() => { jasmine.addMatchers(customMatchers); }); - it('should have all the steps for meta alerts workflow', () => { + it('should have all the steps for meta alerts workflow', async function() : Promise { let comment1 = 'This is a sample comment'; let userNameAndTimestamp = '- admin - a few seconds ago'; -let confirmText = 'Do you wish to create a meta alert with 113 selected alerts?'; +let confirmText = 'Do you wish to create a meta alert with 13 selected alerts?'; let dashRowValues = { - 'firstDashRow': ['0', '192.168.138.158', 'ALERTS', '113'], - 'secondDashRow': ['0', '192.168.66.1', 'ALERTS', '56'] + 'firstDashRow': ['0', 'runlove.us', 'ALERTS', '13'] }; -tablePage.navigateTo(); +await tablePage.navigateTo(); +expect(await tablePage.getChangesAlertTableTitle('Alerts (0)')).toEqual('Alerts (169)'); /* Create Meta Alert */ -treePage.selectGroup('ip_src_addr'); - expect(treePage.getDashGroupValues('192.168.138.158')).toEqualBcoz(dashRowValues.firstDashRow, 'First Dashrow to be present'); - expect(treePage.getDashGroupValues('192.168.66.1')).toEqualBcoz(dashRowValues.secondDashRow, 'Second Dashrow to be present'); +await treePage.selectGroup('host'); +expect(await treePage.getDashGroupValues('runlove.us')).toEqualBcoz(dashRowValues.firstDashRow, 'First Dashrow to be present'); -treePage.clickOnMergeAlerts('192.168.138.158'); -expect(treePage.getConfirmationText()).toEqualBcoz(confirmText, 'confirmation text to be present'); -treePage.clickNoForConfirmation(); +await treePage.clickOnMergeAlerts('runlove.us'); +expect(await treePage.getConfirmationText()).toEqualBcoz(confirmText, 'confirmation text to be present'); +await treePage.clickNoForConfirmation(); -treePage.clickOnMergeAlerts('192.168.138.158'); -treePage.clickYesForConfirmation(); +await treePage.clickOnMergeAlerts('runlove.us'); +await treePage.clickYesForConfirmation(); -treePage.waitForElementToDisappear('192.168.138.158'); +await treePage.waitForElementToDisappear('runlove.us'); -treePage.unGroup(); +await treePage.unGroup(); /* Table should have all alerts */ -tablePage.waitForMetaAlert(); -expect(tablePage.getPaginationText()).toEqualBcoz('1 - 25 of 57', 'pagination text to be present'); -expect(tablePage.getCellValue(0, 2, '(114)')).toContain('(113)', 'number of alerts in a meta alert should be cor
[jira] [Commented] (METRON-1489) Retrofit UI tests to run reliably during nightly QE runs
[ https://issues.apache.org/jira/browse/METRON-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474187#comment-16474187 ] ASF GitHub Bot commented on METRON-1489: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1004 Please fix the formatting issue with the README. I have mentioned in another thread, but you might have missed it. > Retrofit UI tests to run reliably during nightly QE runs > > > Key: METRON-1489 > URL: https://issues.apache.org/jira/browse/METRON-1489 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1004: METRON-1489: Retrofit UI tests to run reliably
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1004 Please fix the formatting issue with the README. I have mentioned in another thread, but you might have missed it. ---
[jira] [Created] (METRON-1554) Pcap Query Panel
Ryan Merriman created METRON-1554: - Summary: Pcap Query Panel Key: METRON-1554 URL: https://issues.apache.org/jira/browse/METRON-1554 Project: Metron Issue Type: New Feature Reporter: Ryan Merriman Legacy OpenSOC included a panel in Kibana that allowed users to query for pcap data. We would like to add this feature back into Metron. There are 2 discussions happening on the dev list where we are gathering user requirements: [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3CCAEVkqPYxfe3Q65mX7Mkuk_FKUCV420yb6hcLmf+FF=1ozer...@mail.gmail.com%3E] and working through the backend architecture: [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3ccaevkqpbxzjnu_wgrbfwnz-mvqnkb7mthedveq9plyhwfit7...@mail.gmail.com%3E] Forthcoming sub tasks will be based on the outcome of these discussions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1554) Pcap Query Panel
[ https://issues.apache.org/jira/browse/METRON-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474247#comment-16474247 ] Otto Fowler commented on METRON-1554: - [~rmerriman] are there any materials around the old functionality that we can reference? > Pcap Query Panel > > > Key: METRON-1554 > URL: https://issues.apache.org/jira/browse/METRON-1554 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Priority: Major > > Legacy OpenSOC included a panel in Kibana that allowed users to query for > pcap data. We would like to add this feature back into Metron. There are 2 > discussions happening on the dev list where we are gathering user > requirements: > [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3CCAEVkqPYxfe3Q65mX7Mkuk_FKUCV420yb6hcLmf+FF=1ozer...@mail.gmail.com%3E] > and working through the backend architecture: > [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3ccaevkqpbxzjnu_wgrbfwnz-mvqnkb7mthedveq9plyhwfit7...@mail.gmail.com%3E] > Forthcoming sub tasks will be based on the outcome of these discussions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1004: METRON-1489: Retrofit UI tests to run reliably
Github user sardell commented on a diff in the pull request: https://github.com/apache/metron/pull/1004#discussion_r187971664 --- Diff: metron-interface/metron-alerts/README.md --- @@ -126,7 +126,14 @@ The application will be available at http://host:4201 assuming the port is set t ## E2E Tests -An expressjs server is available for mocking the elastic search api. +#Caveats --- End diff -- Thanks for pointing this out, @nickwallen. Fixed with the latest commit. ---
[jira] [Commented] (METRON-1489) Retrofit UI tests to run reliably during nightly QE runs
[ https://issues.apache.org/jira/browse/METRON-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474269#comment-16474269 ] ASF GitHub Bot commented on METRON-1489: Github user sardell commented on a diff in the pull request: https://github.com/apache/metron/pull/1004#discussion_r187971664 --- Diff: metron-interface/metron-alerts/README.md --- @@ -126,7 +126,14 @@ The application will be available at http://host:4201 assuming the port is set t ## E2E Tests -An expressjs server is available for mocking the elastic search api. +#Caveats --- End diff -- Thanks for pointing this out, @nickwallen. Fixed with the latest commit. > Retrofit UI tests to run reliably during nightly QE runs > > > Key: METRON-1489 > URL: https://issues.apache.org/jira/browse/METRON-1489 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1012: METRON-1551 Profiler Should Not Use Java Serialization
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1012 Ok, this looks great, one request though. It appears that, at least for the profiler, it is required that objects used in Stellar be Java serializable if they aren't listed in that exception list. We ensure that the objects are kryo serializable [here](https://github.com/apache/metron/blob/91a017b6dcefc250bfd67cecf9803cb59015d213/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/StellarProcessorUtils.java#L64) Can we also make a blurb that they're java serializable? I know that OnlineStatisticsProvider will not be java serializable due to its dependence on t-digest, so maybe make that check optional but on by default? ---
[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization
[ https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474333#comment-16474333 ] ASF GitHub Bot commented on METRON-1551: Github user cestella commented on the issue: https://github.com/apache/metron/pull/1012 Ok, this looks great, one request though. It appears that, at least for the profiler, it is required that objects used in Stellar be Java serializable if they aren't listed in that exception list. We ensure that the objects are kryo serializable [here](https://github.com/apache/metron/blob/91a017b6dcefc250bfd67cecf9803cb59015d213/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/StellarProcessorUtils.java#L64) Can we also make a blurb that they're java serializable? I know that OnlineStatisticsProvider will not be java serializable due to its dependence on t-digest, so maybe make that check optional but on by default? > Profiler Should Not Use Java Serialization > -- > > Key: METRON-1551 > URL: https://issues.apache.org/jira/browse/METRON-1551 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > When running the Profiler in a topology where serialization occurs, the > following error happens. This can occur when the number of workers is > greater than 1. > The topology should not be using Java serialization for serializing tuple > values as this will negatively impact performance. > {code} > 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) > [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] > Caused by: java.lang.RuntimeException: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) > ~[kryo-3.0.3.jar:?] > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) > ~[kryo-3.0.3.jar:?] > at > org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > ... 6 more > Caused by: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > ~[?:1.8.0_162] >
[jira] [Created] (METRON-1555) Update REST to run YARN and MR jobs
Ryan Merriman created METRON-1555: - Summary: Update REST to run YARN and MR jobs Key: METRON-1555 URL: https://issues.apache.org/jira/browse/METRON-1555 Project: Metron Issue Type: Sub-task Reporter: Ryan Merriman Assignee: Ryan Merriman This task involves enabling REST to submit YARN or MR jobs. We will likely need to: * update Maven dependencies to include YARN and MR libraries in the classpath and resolve any version conflicts * update REST start script to include properties required for YARN * update the MPack for any additional setup work (create user HDFS directory for example) and properties needed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1556) Create YARN service
Ryan Merriman created METRON-1556: - Summary: Create YARN service Key: METRON-1556 URL: https://issues.apache.org/jira/browse/METRON-1556 Project: Metron Issue Type: Sub-task Reporter: Ryan Merriman We need a YARN Service in REST for interacting with the YARN REST api. The service should at least: * Return a description of a job including status, start time, end time, and any other relevant information given a job id * Stop a job given a job id The [StormStatusService|https://github.com/apache/metron/blob/master/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java] is similar in that it also proxies to a REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1557) PcapJob should be asynchronous
Ryan Merriman created METRON-1557: - Summary: PcapJob should be asynchronous Key: METRON-1557 URL: https://issues.apache.org/jira/browse/METRON-1557 Project: Metron Issue Type: Sub-task Reporter: Ryan Merriman Pcap job submission should be asynchronous. The PcapJob class should submit the job and return the job id as soon as possible rather than blocking and waiting on the job to complete. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1558) Enable paging through Pcap result sets
Ryan Merriman created METRON-1558: - Summary: Enable paging through Pcap result sets Key: METRON-1558 URL: https://issues.apache.org/jira/browse/METRON-1558 Project: Metron Issue Type: Sub-task Reporter: Ryan Merriman Due to the volume of pcap data and the potential for very large query results, the Pcap query process should write results in manageable chunks or files, so that a user can page through the results in an efficient manner. Page size should be configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1012: METRON-1551 Profiler Should Not Use Java Serialization
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1012 I made two updates. * I added the additional test check to `StellarProcessorUtils` to ensure the value returned is Java serializable. There were a few classes that I had to update for the tests to continue to pass. All simple cases where a class just needed to implement Serializable. * Updated all Profiler classes to be Java serializable in case a user chooses to use Java serialization in their Storm topology. This is not recommended, but there is no reason to block a user from doing so. ---
[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization
[ https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474572#comment-16474572 ] ASF GitHub Bot commented on METRON-1551: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1012 I made two updates. * I added the additional test check to `StellarProcessorUtils` to ensure the value returned is Java serializable. There were a few classes that I had to update for the tests to continue to pass. All simple cases where a class just needed to implement Serializable. * Updated all Profiler classes to be Java serializable in case a user chooses to use Java serialization in their Storm topology. This is not recommended, but there is no reason to block a user from doing so. > Profiler Should Not Use Java Serialization > -- > > Key: METRON-1551 > URL: https://issues.apache.org/jira/browse/METRON-1551 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > When running the Profiler in a topology where serialization occurs, the > following error happens. This can occur when the number of workers is > greater than 1. > The topology should not be using Java serialization for serializing tuple > values as this will negatively impact performance. > {code} > 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) > [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] > Caused by: java.lang.RuntimeException: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) > ~[kryo-3.0.3.jar:?] > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) > ~[kryo-3.0.3.jar:?] > at > org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > ... 6 more > Caused by: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > ~[?:1.8.0_162] > at > org.apache.storm.serialization.Serializab
[jira] [Created] (METRON-1559) Create Pcap Service
Ryan Merriman created METRON-1559: - Summary: Create Pcap Service Key: METRON-1559 URL: https://issues.apache.org/jira/browse/METRON-1559 Project: Metron Issue Type: Sub-task Reporter: Ryan Merriman We need a service that exposes the various Pcap endpoints. These include: * GET /api/v1/pcap/metadata?basePath - This endpoint will return metadata of pcap data stored in HDFS. This would include pcap size, date ranges (how far back can I go), etc. It would accept an optional HDFS basePath parameter for cases where pcap data is stored in multiple places and/or different from the default location. * POST /api/v1/pcap/fixed - This endpoint would accept a fixed pcap request, submit a pcap job, and return a job id. The request would be an object containing the options documented here for the fixed filter: [https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#query-filter-utility]. A job will be associated with a user that submits it. An exception will be returned for violating constraints like too many queries submitted, query parameters out of limits, etc. A record of the user and job id will be persisted to a data store so a list of a user's jobs can later be retrieved. * POST /api/v1/pcap/query - This endpoint would accept a query pcap request, submit a pcap job, and return a job id. The request would be an object containing the options documented here for the query filter: [https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#query-filter-utility]. A job will be associated with a user that submits it. An exception will be returned for violating constraints like too many queries submitted, query parameters out of limits, etc. A record of the user and job id will be persisted to a data store so a list of a user's jobs can later be retrieved. * GET /api/v1/pcap/status/ - This endpoint will return the YARN status of a running/completed job. * GET /api/v1/pcap/stop/ - This endpoint would kill a running pcap job. If the job has already completed this is a noop. * GET /api/v1/pcap/list - This endpoint will list a user's submitted pcap queries. Items in the list would contain job id, status (is it finished?), start/end time, and number of pages. * GET /api/v1/pcap/pdml// - This endpoint will return pcap results for the given page in pdml format ([https://wiki.wireshark.org/PDML]). Are there other formats we want to support? * GET /api/v1/pcap/raw// - This endpoint will allow a user to download raw pcap results for the given page. * DELETE /api/v1/pcap/ - This endpoint will delete pcap query results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1560) Update MPack to support Pcap panel
Ryan Merriman created METRON-1560: - Summary: Update MPack to support Pcap panel Key: METRON-1560 URL: https://issues.apache.org/jira/browse/METRON-1560 Project: Metron Issue Type: Sub-task Reporter: Ryan Merriman The MPack will need to be adjusted to support the Pcap panel. This includes: * prompting a user during the MPack installation process to agree to tshark being installed * installing tshark on the correct node(s) * adding pcap related properties to the MPack so they can be managed in Ambari -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1561) Add security to Pcap endpoints
Ryan Merriman created METRON-1561: - Summary: Add security to Pcap endpoints Key: METRON-1561 URL: https://issues.apache.org/jira/browse/METRON-1561 Project: Metron Issue Type: Sub-task Reporter: Ryan Merriman Pcap endpoints should be secured as follows: * users should only be able to view status and access/cleanup results of the queries they submit * an admin role should be provided that has access to and can perform actions on all jobs/queries * a map of users and jobs they have submitted should be persisted so that a list of a user's jobs can be returned -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1562) Enable Kerberos in REST for YARN and MR jobs
Ryan Merriman created METRON-1562: - Summary: Enable Kerberos in REST for YARN and MR jobs Key: METRON-1562 URL: https://issues.apache.org/jira/browse/METRON-1562 Project: Metron Issue Type: Sub-task Reporter: Ryan Merriman We need to enable Kerberos support in REST for YARN and MR interactions. This will include: * Kerberos authentication for YARN REST api * Kerberos authentication for submitting MR jobs * Kerberos authentication for accessing query results (this is likely already done but should be validated) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1554) Pcap Query Panel
[ https://issues.apache.org/jira/browse/METRON-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474591#comment-16474591 ] Ryan Merriman commented on METRON-1554: --- [~ottobackwards] I believe a lot of the functionality was carried forward and is contained in these READMEs: * [https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend|https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend,] * [https://github.com/apache/metron/tree/master/metron-platform/metron-api] As far as the old UI functionality, the code is spread out in various source code files under [https://github.com/apache/metron/tree/Metron_0.1BETA/metron-ui.] I can't find any documentation on it though. > Pcap Query Panel > > > Key: METRON-1554 > URL: https://issues.apache.org/jira/browse/METRON-1554 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Priority: Major > > Legacy OpenSOC included a panel in Kibana that allowed users to query for > pcap data. We would like to add this feature back into Metron. There are 2 > discussions happening on the dev list where we are gathering user > requirements: > [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3CCAEVkqPYxfe3Q65mX7Mkuk_FKUCV420yb6hcLmf+FF=1ozer...@mail.gmail.com%3E] > and working through the backend architecture: > [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3ccaevkqpbxzjnu_wgrbfwnz-mvqnkb7mthedveq9plyhwfit7...@mail.gmail.com%3E] > Forthcoming sub tasks will be based on the outcome of these discussions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1012: METRON-1551 Profiler Should Not Use Java Serialization
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1012 Looks good; +1 by inspection pending travis ---
[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization
[ https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474636#comment-16474636 ] ASF GitHub Bot commented on METRON-1551: Github user cestella commented on the issue: https://github.com/apache/metron/pull/1012 Looks good; +1 by inspection pending travis > Profiler Should Not Use Java Serialization > -- > > Key: METRON-1551 > URL: https://issues.apache.org/jira/browse/METRON-1551 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > When running the Profiler in a topology where serialization occurs, the > following error happens. This can occur when the number of workers is > greater than 1. > The topology should not be using Java serialization for serializing tuple > values as this will negatively impact performance. > {code} > 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) > [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] > Caused by: java.lang.RuntimeException: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) > ~[kryo-3.0.3.jar:?] > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) > ~[kryo-3.0.3.jar:?] > at > org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > ... 6 more > Caused by: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > ~[?:1.8.0_162] > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) > ~[kryo-3.0.3.jar:?] > at com.esotericsoftware.kryo.Kryo.
[GitHub] metron issue #1012: METRON-1551 Profiler Should Not Use Java Serialization
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1012 Reopen for unrelated failure. Will open a bug for what seems to be an intermittent test failure. ``` Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec <<< FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest) Time elapsed: 0.031 sec <<< FAILURE! java.lang.AssertionError: expected:<6> but was:<5> ``` ---
[GitHub] metron pull request #1012: METRON-1551 Profiler Should Not Use Java Serializ...
GitHub user nickwallen reopened a pull request: https://github.com/apache/metron/pull/1012 METRON-1551 Profiler Should Not Use Java Serialization When running the Profiler in a topology where serialization occurs, the following error happens. This can occur when the number of workers is greater than 1. The topology should not be using Java serialization for serializing tuple values as this will negatively impact performance. ``` 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] Caused by: java.lang.RuntimeException: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult at org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?] at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] ... 6 more Caused by: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) ~[?:1.8.0_162] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[?:1.8.0_162] at org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?] at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_B
[GitHub] metron pull request #1012: METRON-1551 Profiler Should Not Use Java Serializ...
Github user nickwallen closed the pull request at: https://github.com/apache/metron/pull/1012 ---
[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization
[ https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474649#comment-16474649 ] ASF GitHub Bot commented on METRON-1551: GitHub user nickwallen reopened a pull request: https://github.com/apache/metron/pull/1012 METRON-1551 Profiler Should Not Use Java Serialization When running the Profiler in a topology where serialization occurs, the following error happens. This can occur when the number of workers is greater than 1. The topology should not be using Java serialization for serializing tuple values as this will negatively impact performance. ``` 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] Caused by: java.lang.RuntimeException: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult at org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?] at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] ... 6 more Caused by: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) ~[?:1.8.0_162] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[?:1.8.0_162] at org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?] at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) ~[storm-core-1.1.0.2.6.4.
[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization
[ https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474648#comment-16474648 ] ASF GitHub Bot commented on METRON-1551: Github user nickwallen closed the pull request at: https://github.com/apache/metron/pull/1012 > Profiler Should Not Use Java Serialization > -- > > Key: METRON-1551 > URL: https://issues.apache.org/jira/browse/METRON-1551 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > When running the Profiler in a topology where serialization occurs, the > following error happens. This can occur when the number of workers is > greater than 1. > The topology should not be using Java serialization for serializing tuple > values as this will negatively impact performance. > {code} > 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) > [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] > Caused by: java.lang.RuntimeException: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) > ~[kryo-3.0.3.jar:?] > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) > ~[kryo-3.0.3.jar:?] > at > org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > ... 6 more > Caused by: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > ~[?:1.8.0_162] > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) > ~[kryo-3.0.3.jar:?] > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) > ~[kryo-3.0.3.ja
[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization
[ https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474647#comment-16474647 ] ASF GitHub Bot commented on METRON-1551: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1012 Reopen for unrelated failure. Will open a bug for what seems to be an intermittent test failure. ``` Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec <<< FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest) Time elapsed: 0.031 sec <<< FAILURE! java.lang.AssertionError: expected:<6> but was:<5> ``` > Profiler Should Not Use Java Serialization > -- > > Key: METRON-1551 > URL: https://issues.apache.org/jira/browse/METRON-1551 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > When running the Profiler in a topology where serialization occurs, the > following error happens. This can occur when the number of workers is > greater than 1. > The topology should not be using Java serialization for serializing tuple > values as this will negatively impact performance. > {code} > 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) > [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] > Caused by: java.lang.RuntimeException: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) > ~[kryo-3.0.3.jar:?] > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) > ~[kryo-3.0.3.jar:?] > at > org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > ... 6 more > Caused by: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > ~[?:1.8.0_162] > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1
[jira] [Created] (METRON-1563) Initial Feature Branch Work
Otto Fowler created METRON-1563: --- Summary: Initial Feature Branch Work Key: METRON-1563 URL: https://issues.apache.org/jira/browse/METRON-1563 Project: Metron Issue Type: Sub-task Reporter: Otto Fowler Assignee: Otto Fowler -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #687: METRON-1563 [DISCUSS] Add Assignment to Stellar Language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/687 I have renamed in preparation for the Feature branch based on the original jira ( create a new subtask to land this on feature and change to that ID ) ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474687#comment-16474687 ] ASF GitHub Bot commented on METRON-1563: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/687 I have renamed in preparation for the Feature branch based on the original jira ( create a new subtask to land this on feature and change to that ID ) > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #973: METRON-1356: Add a mechanism in Java for discovering serv...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/973 @ottobackwards just updated the README. You previously had to lookup the docker machine ip address and substitute it in source code to get the tests to pass if running locally. The text I removed in the latest commit described how to do that. So for testing this, you should no longer have to do that and the tests should just work. I haven't heard of that before but I will check it out. ---
[jira] [Commented] (METRON-1356) Add a mechanism in Java for discovering service host/ports
[ https://issues.apache.org/jira/browse/METRON-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474716#comment-16474716 ] ASF GitHub Bot commented on METRON-1356: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/973 @ottobackwards just updated the README. You previously had to lookup the docker machine ip address and substitute it in source code to get the tests to pass if running locally. The text I removed in the latest commit described how to do that. So for testing this, you should no longer have to do that and the tests should just work. I haven't heard of that before but I will check it out. > Add a mechanism in Java for discovering service host/ports > -- > > Key: METRON-1356 > URL: https://issues.apache.org/jira/browse/METRON-1356 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Priority: Major > > Integration tests will need to initialize clients with service urls. These > may change depending on where and how the infrastructure is run (Docker > engine vs Docker for Mac). It would be helpful to have a unified way of > retrieving these across all integration tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1010: METRON-1548: Remove hardcoded source:type from Al...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1010#discussion_r188086075 --- Diff: metron-interface/metron-alerts/src/app/service/global-config.service.ts --- @@ -0,0 +1,88 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import {Injectable, Inject} from '@angular/core'; +import {Http, Headers, RequestOptions, Response, ResponseOptions} from '@angular/http'; +import {Observable} from 'rxjs/Observable'; +import {HttpUtil} from '../utils/httpUtil'; + +@Injectable() +export class GlobalConfigService { + url = 'api/v1/global/config'; + defaultHeaders = {'Content-Type': 'application/json', 'X-Requested-With': 'XMLHttpRequest'}; + + private globalConfig = {}; + + constructor(private http: Http) { --- End diff -- I think this constructor was carried over from metron-config and the property assignments are not needed. I think it should just be empty. ---
[jira] [Commented] (METRON-1548) Alerts UI: Remove hardcoded source:type and other fields that may vary
[ https://issues.apache.org/jira/browse/METRON-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474753#comment-16474753 ] ASF GitHub Bot commented on METRON-1548: Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1010#discussion_r188086075 --- Diff: metron-interface/metron-alerts/src/app/service/global-config.service.ts --- @@ -0,0 +1,88 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import {Injectable, Inject} from '@angular/core'; +import {Http, Headers, RequestOptions, Response, ResponseOptions} from '@angular/http'; +import {Observable} from 'rxjs/Observable'; +import {HttpUtil} from '../utils/httpUtil'; + +@Injectable() +export class GlobalConfigService { + url = 'api/v1/global/config'; + defaultHeaders = {'Content-Type': 'application/json', 'X-Requested-With': 'XMLHttpRequest'}; + + private globalConfig = {}; + + constructor(private http: Http) { --- End diff -- I think this constructor was carried over from metron-config and the property assignments are not needed. I think it should just be empty. > Alerts UI: Remove hardcoded source:type and other fields that may vary > -- > > Key: METRON-1548 > URL: https://issues.apache.org/jira/browse/METRON-1548 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Priority: Major > > In Solr, we use source.type instead of source:type (which was originally due > to ES limitations, we'd prefer it to be common across both). However, it's > hardcoded in the alerts UI to be source:type, so nothing in the UI has a > source type and it breaks other things like metaalerts (which rely on source > type for being able to pull the data together). > Any other configs / queries that would cause similar problems should be > updated appropriately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1011: METRON-1552: Add gzip file validation check to th...
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/1011#discussion_r188093107 --- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java --- @@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) throws IOException { System.out.println("Successfully created and updated new GeoIP information"); } - protected File downloadGeoFile(String urlStr, String tmpDir) { + protected File downloadGeoFile(String urlStr, String tmpDir, int numRetries) { File localFile = null; -try { - URL url = new URL(urlStr); - localFile = new File(tmpDir + new File(url.getPath()).getName()); +int attempts = 0; +boolean valid = false; +while (attempts <= numRetries) { + try { +URL url = new URL(urlStr); +localFile = new File(tmpDir + new File(url.getPath()).getName()); - System.out.println("Downloading " + url.toString() + " to " + localFile.getAbsolutePath()); - if (localFile.exists() && !localFile.delete()) { -System.err.println("File already exists locally and can't be deleted. Please delete before continuing"); -System.exit(3); +System.out.println("Downloading " + url.toString() + " to " + localFile.getAbsolutePath()); +if (localFile.exists() && !localFile.delete()) { + System.err.println( + "File already exists locally and can't be deleted. Please delete before continuing"); + System.exit(3); +} +FileUtils.copyURLToFile(url, localFile, 5000, 1); +if (!CompressionStrategies.GZIP.test(localFile)) { + throw new IOException("Invalid Gzip file"); +} + } catch (MalformedURLException e) { +System.err.println("Malformed URL - aborting: " + e); +e.printStackTrace(); +System.exit(4); --- End diff -- The existing exit call seemed reasonable to me if only because I'd expect this to be user error, not due to issues with an external system. ---
[jira] [Commented] (METRON-1552) Add gzip file validation check to the geo loader
[ https://issues.apache.org/jira/browse/METRON-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474783#comment-16474783 ] ASF GitHub Bot commented on METRON-1552: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/1011#discussion_r188093107 --- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java --- @@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) throws IOException { System.out.println("Successfully created and updated new GeoIP information"); } - protected File downloadGeoFile(String urlStr, String tmpDir) { + protected File downloadGeoFile(String urlStr, String tmpDir, int numRetries) { File localFile = null; -try { - URL url = new URL(urlStr); - localFile = new File(tmpDir + new File(url.getPath()).getName()); +int attempts = 0; +boolean valid = false; +while (attempts <= numRetries) { + try { +URL url = new URL(urlStr); +localFile = new File(tmpDir + new File(url.getPath()).getName()); - System.out.println("Downloading " + url.toString() + " to " + localFile.getAbsolutePath()); - if (localFile.exists() && !localFile.delete()) { -System.err.println("File already exists locally and can't be deleted. Please delete before continuing"); -System.exit(3); +System.out.println("Downloading " + url.toString() + " to " + localFile.getAbsolutePath()); +if (localFile.exists() && !localFile.delete()) { + System.err.println( + "File already exists locally and can't be deleted. Please delete before continuing"); + System.exit(3); +} +FileUtils.copyURLToFile(url, localFile, 5000, 1); +if (!CompressionStrategies.GZIP.test(localFile)) { + throw new IOException("Invalid Gzip file"); +} + } catch (MalformedURLException e) { +System.err.println("Malformed URL - aborting: " + e); +e.printStackTrace(); +System.exit(4); --- End diff -- The existing exit call seemed reasonable to me if only because I'd expect this to be user error, not due to issues with an external system. > Add gzip file validation check to the geo loader > > > Key: METRON-1552 > URL: https://issues.apache.org/jira/browse/METRON-1552 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1012: METRON-1551 Profiler Should Not Use Java Serializ...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1012 ---
[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization
[ https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474796#comment-16474796 ] ASF GitHub Bot commented on METRON-1551: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1012 > Profiler Should Not Use Java Serialization > -- > > Key: METRON-1551 > URL: https://issues.apache.org/jira/browse/METRON-1551 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > When running the Profiler in a topology where serialization occurs, the > following error happens. This can occur when the number of workers is > greater than 1. > The topology should not be using Java serialization for serializing tuple > values as this will negatively impact performance. > {code} > 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) > [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] > Caused by: java.lang.RuntimeException: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) > ~[kryo-3.0.3.jar:?] > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) > ~[kryo-3.0.3.jar:?] > at > org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > ... 6 more > Caused by: java.io.NotSerializableException: > org.apache.metron.common.configuration.profiler.ProfileResult > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > ~[?:1.8.0_162] > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > ~[?:1.8.0_162] > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > ~[?:1.8.0_162] > at > org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38) > ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) > ~[kryo-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) > ~[kryo-3.0.3.jar:?] > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) > ~[kryo-3.0.3.jar:?]
[GitHub] metron pull request #687: METRON-1563 [DISCUSS] Add Assignment to Stellar La...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/687 ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474805#comment-16474805 ] ASF GitHub Bot commented on METRON-1563: Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/687 > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1014: METRON-1563 : Base Stellar assign for feature bra...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1014 METRON-1563 : Base Stellar assign for feature branch repackage: https://github.com/apache/metron/pull/687 Please sanity check and see that PR You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar_assign Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1014.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1014 commit 37587689e9feadeb54465aa7319f114a4e11255b Author: cstella Date: 2017-12-12T20:24:07Z METRON-1306: When index template install fails, we should fail the install closes apache/incubator-metron#834 commit 6ca08b9f598b649d60c6b6468164a374b7f6f555 Author: MohanDV Date: 2017-12-13T15:55:04Z METRON-1343 Swagger UI for User Controller needs request method (MohanDV via ottobackwards) closes apache/metron#862 commit d9ed1bad1f5b2e2471fbea11353f2947f7f52e13 Author: nickwallen Date: 2017-12-14T15:59:27Z METRON-1349 Full Dev Builds Metron Twice (nickwallen) closes apache/metron#866 commit c4cee6af64eda4db4da3eff86abab7d4ae4ec56a Author: mmiklavc Date: 2017-12-14T20:29:49Z METRON-1345: Update EC2 README for custom Ansible (mmiklavc via mmiklavc) closes apache/metron#859 commit d446da8e707b1069576b049452484e088a3eeede Author: nickwallen Date: 2017-12-19T17:42:39Z METRON-1372 Validate JIRA for Releases (nickwallen) closes apache/metron#874 commit adb024070d5d098909fb3800875d0042aeb27c92 Author: ottobackwards Date: 2017-12-19T20:13:22Z METRON-1374 Script the RC checking process (ottobackwards) closes apache/metron#876 commit 3f0b1b7b4a002d3f364bd2aee7b5921c0435c4a4 Author: cstella Date: 2017-12-20T14:30:03Z METRON-1350: Add reservoir sampling functions to Stellar closes apache/incubator-metron#867 commit 196da12c43337d019a52b99bf6178fbda45f886d Author: nickwallen Date: 2017-12-21T14:04:49Z METRON-1348 Metron Service Checks Use Wrong Hostname (nickwallen) closes apache/metron#864 commit 76bed5d754fcf358809f0be7a034758b9b20fc5e Author: cstella Date: 2017-12-21T21:49:31Z METRON-1365: Allow PROFILE_GET to return a default value for a profile and entity that does not have a value written. closes apache/incubator-metron#871 commit 3612a89216bd57c40a1bc3e27853c6146b471e1e Author: ottobackwards Date: 2017-12-25T20:44:45Z METRON-1376 RC Check Script should have named parameters (ottobackwards via nickwallen) closes apache/metron#877 commit fc8723e461d655e315d0b51acd1a31f82b4efd1f Author: nickwallen Date: 2017-12-27T18:25:53Z METRON-1351 Create Installable Packages for Ubuntu Trusty (nickwallen) closes apache/metron#868 commit 0518408513ed54df8dbe234027b353bed2e61943 Author: mattf-horton Date: 2017-12-31T22:01:29Z METRON-1373 RAT failure for metron-interface/metron-alerts (mattf-horton) closes apache/metron#875 commit 3b10f84cc49993a1c5917f54be6ca313c8d780c4 Author: justinleet Date: 2018-01-01T23:49:57Z METRON-1071 Create CONTRIBUTING.md (justinleet) closes apache/metron#881 commit 2d9d7a5f6302267edafba772268145e76751795c Author: justinleet Date: 2018-01-02T16:38:54Z METRON-1381 Add Apache license to MD files and remove the Rat exclusion (justinleet) closes apache/metron#883 commit 8a61b96b6b01b247a8ff8d800730378b8da23471 Author: mattf-horton Date: 2018-01-02T17:55:52Z METRON-1384 Increment master version number to 0.4.3 for on-going development (mattf-horton via nickwallen) closes apache/metron#885 commit 01c26a77b1041204b0bbbc544cc0a5d02e9339a8 Author: nickwallen Date: 2018-01-03T14:52:57Z METRON-1362 Improve Metron Deployment README (nickwallen) closes apache/metron#869 commit 3381b853dca1c08a7a083593045dec2c7d4d92db Author: mattf-horton Date: 2018-01-04T20:30:30Z METRON-1388 update public web site to point at 0.4.2 new release (mattf-horton) closes apache/metron#887 commit 9108072756b6ffeedade985d3cd52ef7338cd61a Author: merrimanr Date: 2018-01-08T14:18:30Z METRON-1385 Missing "properties" in index template causes ElasticsearchColumnMetadataDao.getColumnMetadata to fail (merrimanr) closes apache/metron#886 commit 0996b7348eca14fea1b1b3c4dd57861b3a30bdeb Author: cstella Date: 2018-01-08T14:30:58Z METRON-1377: Stellar function to generate typosquatted domains (similar to dnstwist) closes apache/incubator-metron#878 commit e821391895f0f3523b4e05ae5d639c13b0113855 Author: mmiklavc Date: 2018-01-08T19:08:40Z METRON-939: Upgrade ElasticSearch and Kibana (mmiklavc via mmiklavc) closes apache/metron#840 commit c559ed7e1838ec71344eae3d9e37771db2641635 Author: cstella Date: 2018-01-09T20:28:47Z METRON-1379: Add an OBJECT_GET stellar function closes apache/inc
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 @cestella I rebased this on the new feature branch ( after rebasing on the same master ) and I get all of these other commits. I don't know how to get rid of them? ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474822#comment-16474822 ] ASF GitHub Bot commented on METRON-1563: GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/1014 METRON-1563 : Base Stellar assign for feature branch repackage: https://github.com/apache/metron/pull/687 Please sanity check and see that PR You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar_assign Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1014.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1014 commit 37587689e9feadeb54465aa7319f114a4e11255b Author: cstella Date: 2017-12-12T20:24:07Z METRON-1306: When index template install fails, we should fail the install closes apache/incubator-metron#834 commit 6ca08b9f598b649d60c6b6468164a374b7f6f555 Author: MohanDV Date: 2017-12-13T15:55:04Z METRON-1343 Swagger UI for User Controller needs request method (MohanDV via ottobackwards) closes apache/metron#862 commit d9ed1bad1f5b2e2471fbea11353f2947f7f52e13 Author: nickwallen Date: 2017-12-14T15:59:27Z METRON-1349 Full Dev Builds Metron Twice (nickwallen) closes apache/metron#866 commit c4cee6af64eda4db4da3eff86abab7d4ae4ec56a Author: mmiklavc Date: 2017-12-14T20:29:49Z METRON-1345: Update EC2 README for custom Ansible (mmiklavc via mmiklavc) closes apache/metron#859 commit d446da8e707b1069576b049452484e088a3eeede Author: nickwallen Date: 2017-12-19T17:42:39Z METRON-1372 Validate JIRA for Releases (nickwallen) closes apache/metron#874 commit adb024070d5d098909fb3800875d0042aeb27c92 Author: ottobackwards Date: 2017-12-19T20:13:22Z METRON-1374 Script the RC checking process (ottobackwards) closes apache/metron#876 commit 3f0b1b7b4a002d3f364bd2aee7b5921c0435c4a4 Author: cstella Date: 2017-12-20T14:30:03Z METRON-1350: Add reservoir sampling functions to Stellar closes apache/incubator-metron#867 commit 196da12c43337d019a52b99bf6178fbda45f886d Author: nickwallen Date: 2017-12-21T14:04:49Z METRON-1348 Metron Service Checks Use Wrong Hostname (nickwallen) closes apache/metron#864 commit 76bed5d754fcf358809f0be7a034758b9b20fc5e Author: cstella Date: 2017-12-21T21:49:31Z METRON-1365: Allow PROFILE_GET to return a default value for a profile and entity that does not have a value written. closes apache/incubator-metron#871 commit 3612a89216bd57c40a1bc3e27853c6146b471e1e Author: ottobackwards Date: 2017-12-25T20:44:45Z METRON-1376 RC Check Script should have named parameters (ottobackwards via nickwallen) closes apache/metron#877 commit fc8723e461d655e315d0b51acd1a31f82b4efd1f Author: nickwallen Date: 2017-12-27T18:25:53Z METRON-1351 Create Installable Packages for Ubuntu Trusty (nickwallen) closes apache/metron#868 commit 0518408513ed54df8dbe234027b353bed2e61943 Author: mattf-horton Date: 2017-12-31T22:01:29Z METRON-1373 RAT failure for metron-interface/metron-alerts (mattf-horton) closes apache/metron#875 commit 3b10f84cc49993a1c5917f54be6ca313c8d780c4 Author: justinleet Date: 2018-01-01T23:49:57Z METRON-1071 Create CONTRIBUTING.md (justinleet) closes apache/metron#881 commit 2d9d7a5f6302267edafba772268145e76751795c Author: justinleet Date: 2018-01-02T16:38:54Z METRON-1381 Add Apache license to MD files and remove the Rat exclusion (justinleet) closes apache/metron#883 commit 8a61b96b6b01b247a8ff8d800730378b8da23471 Author: mattf-horton Date: 2018-01-02T17:55:52Z METRON-1384 Increment master version number to 0.4.3 for on-going development (mattf-horton via nickwallen) closes apache/metron#885 commit 01c26a77b1041204b0bbbc544cc0a5d02e9339a8 Author: nickwallen Date: 2018-01-03T14:52:57Z METRON-1362 Improve Metron Deployment README (nickwallen) closes apache/metron#869 commit 3381b853dca1c08a7a083593045dec2c7d4d92db Author: mattf-horton Date: 2018-01-04T20:30:30Z METRON-1388 update public web site to point at 0.4.2 new release (mattf-horton) closes apache/metron#887 commit 9108072756b6ffeedade985d3cd52ef7338cd61a Author: merrimanr Date: 2018-01-08T14:18:30Z METRON-1385 Missing "properties" in index template causes ElasticsearchColumnMetadataDao.getColumnMetadata to fail (merrimanr) closes apache/metron#886 commit 0996b7348eca14fea1b1b3c4dd57861b3a30bdeb Author: cstella Date: 2018-01-08T14:30:58Z METRON-1377: Stellar function to generate typosquatted domains (similar to dnstwist) closes apache/incubator-metron#878 commit e821391895f0f3523b4e05ae5d639c13b0113855 Author: mmiklavc Date: 2018-01-08T19:08:40Z ME
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474823#comment-16474823 ] ASF GitHub Bot commented on METRON-1563: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 @cestella I rebased this on the new feature branch ( after rebasing on the same master ) and I get all of these other commits. I don't know how to get rid of them? > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #526: Metron-846: Add E2E tests for metron management ui
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/526 I am testing and updating this PR to get it to work on latest master. ---