[GitHub] metron issue #1038: METRON-1586: Defaulting for the source type field in ale...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1038 +1 by inspection and by spinning up full-dev and seeing data in the alerts UI. ---
[jira] [Commented] (METRON-1586) Defaulting for the source type field in alerts UI does not work
[
https://issues.apache.org/jira/browse/METRON-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495930#comment-16495930
]
ASF GitHub Bot commented on METRON-1586:
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/1038
+1 by inspection and by spinning up full-dev and seeing data in the alerts
UI.
> Defaulting for the source type field in alerts UI does not work
> ---
>
> Key: METRON-1586
> URL: https://issues.apache.org/jira/browse/METRON-1586
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Priority: Major
>
> The alerts UI does not allow you to display an individual alert. The error is
> a 404 against the findOne endpoint becuase the sensorType is set to
> "undefined". We should be defaulting this to source:type in the UI.
> The POST data is:
> {guid: "0e0d5f36-0fb5-4348-81fc-a6096ac4f74b", sensorType: "undefined"}
> The core issue is everywhere in
> [https://github.com/apache/metron/pull/1010/files] that we're calling
> this.globalConfig['source.type.field'] should fall-back to source:type
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1039: METRON-1588 Migrate storm-kafka-client to 1.2.1
Github user HeartSaVioR commented on the issue: https://github.com/apache/metron/pull/1039 I've rebased to change commit titles slightly, so the build is re-triggered. The build was succeed before rebasing. Travis build in my folk: https://travis-ci.org/HeartSaVioR/metron/builds/385935022 ---
[jira] [Commented] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[ https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495877#comment-16495877 ] ASF GitHub Bot commented on METRON-1588: Github user HeartSaVioR commented on the issue: https://github.com/apache/metron/pull/1039 I've rebased to change commit titles slightly, so the build is re-triggered. The build was succeed before rebasing. Travis build in my folk: https://travis-ci.org/HeartSaVioR/metron/builds/385935022 > Migrate storm-kafka-client to 1.2.1 > --- > > Key: METRON-1588 > URL: https://issues.apache.org/jira/browse/METRON-1588 > Project: Metron > Issue Type: Improvement >Reporter: Jungtaek Lim >Priority: Critical > > Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says > 1.2.0 to be stabled one, because Storm community resolved 40 issues including > critical and blocker from 1.2.0. > There're still couple of issues after 1.2.0 so better to sync up the version > to the latest, so I suggest Metron to upgrade the version to the latest, > 1.2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[ https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495871#comment-16495871 ] ASF GitHub Bot commented on METRON-1588: Github user HeartSaVioR commented on the issue: https://github.com/apache/metron/pull/1039 I've seen Travis CI succeeded in my fork, but unfortunately I don't know how to do manual test with full dev. Much appreciated if reviewers could do the manual validation phase. > Migrate storm-kafka-client to 1.2.1 > --- > > Key: METRON-1588 > URL: https://issues.apache.org/jira/browse/METRON-1588 > Project: Metron > Issue Type: Improvement >Reporter: Jungtaek Lim >Priority: Critical > > Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says > 1.2.0 to be stabled one, because Storm community resolved 40 issues including > critical and blocker from 1.2.0. > There're still couple of issues after 1.2.0 so better to sync up the version > to the latest, so I suggest Metron to upgrade the version to the latest, > 1.2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1039: METRON-1588 Migrate storm-kafka-client to 1.2.1
Github user HeartSaVioR commented on the issue: https://github.com/apache/metron/pull/1039 I've seen Travis CI succeeded in my fork, but unfortunately I don't know how to do manual test with full dev. Much appreciated if reviewers could do the manual validation phase. ---
[jira] [Commented] (METRON-1586) Defaulting for the source type field in alerts UI does not work
[
https://issues.apache.org/jira/browse/METRON-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495846#comment-16495846
]
ASF GitHub Bot commented on METRON-1586:
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/1038
Correct. I tested with both the field set and unspecified .
> Defaulting for the source type field in alerts UI does not work
> ---
>
> Key: METRON-1586
> URL: https://issues.apache.org/jira/browse/METRON-1586
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Priority: Major
>
> The alerts UI does not allow you to display an individual alert. The error is
> a 404 against the findOne endpoint becuase the sensorType is set to
> "undefined". We should be defaulting this to source:type in the UI.
> The POST data is:
> {guid: "0e0d5f36-0fb5-4348-81fc-a6096ac4f74b", sensorType: "undefined"}
> The core issue is everywhere in
> [https://github.com/apache/metron/pull/1010/files] that we're calling
> this.globalConfig['source.type.field'] should fall-back to source:type
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1038: METRON-1586: Defaulting for the source type field in ale...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1038 Correct. I tested with both the field set and unspecified . ---
[jira] [Commented] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[ https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495838#comment-16495838 ] ASF GitHub Bot commented on METRON-1588: GitHub user HeartSaVioR opened a pull request: https://github.com/apache/metron/pull/1039 METRON-1588 Migrate storm-kafka-client to 1.2.1 ## Contributor Comments Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says 1.2.0 to be stabled one, because Storm community resolved 40 issues including critical and blocker from 1.2.0. There're still couple of issues after 1.2.0 so better to sync up the version to the latest, so I suggest Metron to upgrade the version to the latest, 1.2.1. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/metron METRON-1588-with-relocation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1039.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1039 commit 0382fcc71bdfe29588a2913680f9f9a87e68db16 Author: Jungtaek Lim Date: 2018-05-29T01:01:54Z Upgrade storm-kafka-client to 1.2.1 * this incurs additional transitive dependencies * com.fasterxml.jackson.core:jackson-databind:jar:2.6.3:compile * com.google.guava:guava:jar:16.0.1:compile * commons-lang:commons-lang:jar:2.5:compile commit 056dcabdb634bc6756b52ea4e96fbda1bf0e2e5f Author: Jungtaek Lim Date: 2018-05-29T04:14:35Z Relocate new transitive dependencies from storm-kafka-client commit a586c47f86a1354bc6d8764e44af8750f7de010d Author: Jungtaek Lim Date: 2018-05-29T05:58:09Z Remove unnecessary version properties / conflicted dependency commit 4c43cf0ea39e0ebedefb5c2e31f4940bbec7a696 Author: Jungtaek Lim Date: 2018-05-29T06:45:31Z Fix dependency to avoid adding unnecessary dependency commit baf76f3cb5478ee56ddfc3a5cc301743e317044f Author: Jungtaek Lim Date: 2018-05-29T07:36:01Z Update dependencies > Migrate storm-kafka-client
[GitHub] metron pull request #1039: METRON-1588 Migrate storm-kafka-client to 1.2.1
GitHub user HeartSaVioR opened a pull request: https://github.com/apache/metron/pull/1039 METRON-1588 Migrate storm-kafka-client to 1.2.1 ## Contributor Comments Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says 1.2.0 to be stabled one, because Storm community resolved 40 issues including critical and blocker from 1.2.0. There're still couple of issues after 1.2.0 so better to sync up the version to the latest, so I suggest Metron to upgrade the version to the latest, 1.2.1. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/metron METRON-1588-with-relocation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1039.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1039 commit 0382fcc71bdfe29588a2913680f9f9a87e68db16 Author: Jungtaek Lim Date: 2018-05-29T01:01:54Z Upgrade storm-kafka-client to 1.2.1 * this incurs additional transitive dependencies * com.fasterxml.jackson.core:jackson-databind:jar:2.6.3:compile * com.google.guava:guava:jar:16.0.1:compile * commons-lang:commons-lang:jar:2.5:compile commit 056dcabdb634bc6756b52ea4e96fbda1bf0e2e5f Author: Jungtaek Lim Date: 2018-05-29T04:14:35Z Relocate new transitive dependencies from storm-kafka-client commit a586c47f86a1354bc6d8764e44af8750f7de010d Author: Jungtaek Lim Date: 2018-05-29T05:58:09Z Remove unnecessary version properties / conflicted dependency commit 4c43cf0ea39e0ebedefb5c2e31f4940bbec7a696 Author: Jungtaek Lim Date: 2018-05-29T06:45:31Z Fix dependency to avoid adding unnecessary dependency commit baf76f3cb5478ee56ddfc3a5cc301743e317044f Author: Jungtaek Lim Date: 2018-05-29T07:36:01Z Update dependencies ---
[jira] [Created] (METRON-1588) Migrate storm-kafka-client to 1.2.1
Jungtaek Lim created METRON-1588: Summary: Migrate storm-kafka-client to 1.2.1 Key: METRON-1588 URL: https://issues.apache.org/jira/browse/METRON-1588 Project: Metron Issue Type: Improvement Reporter: Jungtaek Lim Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says 1.2.0 to be stabled one, because Storm community resolved 40 issues including critical and blocker from 1.2.0. There're still couple of issues after 1.2.0 so better to sync up the version to the latest, so I suggest Metron to upgrade the version to the latest, 1.2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1586) Defaulting for the source type field in alerts UI does not work
[
https://issues.apache.org/jira/browse/METRON-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495815#comment-16495815
]
ASF GitHub Bot commented on METRON-1586:
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/1038
Just to be completely clear, this also defaults to `source:type` if the
`source.type.field` is unspecified in the global config, right?
> Defaulting for the source type field in alerts UI does not work
> ---
>
> Key: METRON-1586
> URL: https://issues.apache.org/jira/browse/METRON-1586
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Priority: Major
>
> The alerts UI does not allow you to display an individual alert. The error is
> a 404 against the findOne endpoint becuase the sensorType is set to
> "undefined". We should be defaulting this to source:type in the UI.
> The POST data is:
> {guid: "0e0d5f36-0fb5-4348-81fc-a6096ac4f74b", sensorType: "undefined"}
> The core issue is everywhere in
> [https://github.com/apache/metron/pull/1010/files] that we're calling
> this.globalConfig['source.type.field'] should fall-back to source:type
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1038: METRON-1586: Defaulting for the source type field in ale...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1038 Just to be completely clear, this also defaults to `source:type` if the `source.type.field` is unspecified in the global config, right? ---
[jira] [Commented] (METRON-1586) Defaulting for the source type field in alerts UI does not work
[
https://issues.apache.org/jira/browse/METRON-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495737#comment-16495737
]
ASF GitHub Bot commented on METRON-1586:
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/1038
METRON-1586: Defaulting for the source type field in alerts UI does not work
## Contributor Comments
This PR fixes a bug where the source type field wasn't being properly set
when the setting isn't defined in the global config.
This has been tested in full dev. It can be verified by navigating to the
Alerts UI, logging in, and clicking on a search result in the list of results.
A panel on the right should slide out with the alert details populated. It
should also work if the "source.type.field" is set in the global config to
"source:type" for ES and "source.type" for Solr.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-1586
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1038.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1038
commit a5fdfd3f84fdd81d6e586b9594a0059062f05d5e
Author: merrimanr
Date: 2018-05-30T21:45:49Z
initial commit
> Defaulting for the source type field in alerts UI does not work
> ---
>
> Key: METRON-1586
> URL: https://issues.apache.org/jira/browse/METRON-1586
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Priority: Major
>
> The alerts UI does not allow you to display an individual alert. The error is
> a 404 against the findOne endpoint becuase the sensorType is set to
> "undefined". We should be defaulting this to source:type in the UI.
> The POST data is:
> {guid: "0e0d5f36-0fb5-4348-81fc-a6096ac4f74b", sensorType: "undefined"}
> The core issue is everywhere in
>
[GitHub] metron pull request #1038: METRON-1586: Defaulting for the source type field...
GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1038 METRON-1586: Defaulting for the source type field in alerts UI does not work ## Contributor Comments This PR fixes a bug where the source type field wasn't being properly set when the setting isn't defined in the global config. This has been tested in full dev. It can be verified by navigating to the Alerts UI, logging in, and clicking on a search result in the list of results. A panel on the right should slide out with the alert details populated. It should also work if the "source.type.field" is set in the global config to "source:type" for ES and "source.type" for Solr. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1586 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1038.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1038 commit a5fdfd3f84fdd81d6e586b9594a0059062f05d5e Author: merrimanr Date: 2018-05-30T21:45:49Z initial commit ---
[jira] [Created] (METRON-1587) Make collection utility work for HDP search
Ryan Merriman created METRON-1587: - Summary: Make collection utility work for HDP search Key: METRON-1587 URL: https://issues.apache.org/jira/browse/METRON-1587 Project: Metron Issue Type: Sub-task Reporter: Ryan Merriman Assignee: Ryan Merriman Collection scripts need to be improved so that they use the Solr REST api instead for collection management. Kerberos support should also be included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1586) Defaulting for the source type field in alerts UI does not work
Justin Leet created METRON-1586:
---
Summary: Defaulting for the source type field in alerts UI does
not work
Key: METRON-1586
URL: https://issues.apache.org/jira/browse/METRON-1586
Project: Metron
Issue Type: Bug
Reporter: Justin Leet
The alerts UI does not allow you to display an individual alert. The error is a
404 against the findOne endpoint becuase the sensorType is set to "undefined".
We should be defaulting this to source:type in the UI.
The POST data is:
{guid: "0e0d5f36-0fb5-4348-81fc-a6096ac4f74b", sensorType: "undefined"}
The core issue is everywhere in
[https://github.com/apache/metron/pull/1010/files] that we're calling
this.globalConfig['source.type.field'] should fall-back to source:type
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1032: METRON-1578 - EC2 10 Node Deployment - Reverting to usin...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1032 I'm no Ansible expert by any means, but could we possible leverage something like the following to have it change by client OS instead of hard-coding a commented set of lines? https://ansible-tips-and-tricks.readthedocs.io/en/latest/os-dependent-tasks/variables/ ---
[jira] [Commented] (METRON-1578) EC2 10 Node Deployment - Reverting to using control_path with %C for Mac
[ https://issues.apache.org/jira/browse/METRON-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495538#comment-16495538 ] ASF GitHub Bot commented on METRON-1578: Github user cestella commented on the issue: https://github.com/apache/metron/pull/1032 hrm, I'm not the best qualified to review this, but I wanted to give it a bump so it gets looked at. Any thoughts on this by anyone? > EC2 10 Node Deployment - Reverting to using control_path with %C for Mac > > > Key: METRON-1578 > URL: https://issues.apache.org/jira/browse/METRON-1578 > Project: Metron > Issue Type: Bug >Affects Versions: 0.5.0 > Environment: Mac OS High Sierra >Reporter: ashah >Assignee: ashah >Priority: Minor > Labels: aws, easyfix, newbie > Original Estimate: 1h > Remaining Estimate: 1h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1032: METRON-1578 - EC2 10 Node Deployment - Reverting to usin...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1032 hrm, I'm not the best qualified to review this, but I wanted to give it a bump so it gets looked at. Any thoughts on this by anyone? ---
[jira] [Commented] (METRON-1479) Add editorconfig - create guideline for code formatting in the angular part of the app
[ https://issues.apache.org/jira/browse/METRON-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495436#comment-16495436 ] ASF GitHub Bot commented on METRON-1479: Github user pono closed the pull request at: https://github.com/apache/metron/pull/951 > Add editorconfig - create guideline for code formatting in the angular part > of the app > -- > > Key: METRON-1479 > URL: https://issues.apache.org/jira/browse/METRON-1479 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1478) Move type definitions to dev-dependencies
[ https://issues.apache.org/jira/browse/METRON-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495433#comment-16495433 ] ASF GitHub Bot commented on METRON-1478: Github user pono closed the pull request at: https://github.com/apache/metron/pull/954 > Move type definitions to dev-dependencies > - > > Key: METRON-1478 > URL: https://issues.apache.org/jira/browse/METRON-1478 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend
[ https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495435#comment-16495435 ] ASF GitHub Bot commented on METRON-1480: Github user pono closed the pull request at: https://github.com/apache/metron/pull/952 > Add yarn as default build tool for the frontend > --- > > Key: METRON-1480 > URL: https://issues.apache.org/jira/browse/METRON-1480 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1472) Add stylelint support
[ https://issues.apache.org/jira/browse/METRON-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495434#comment-16495434 ] ASF GitHub Bot commented on METRON-1472: Github user pono closed the pull request at: https://github.com/apache/metron/pull/953 > Add stylelint support > - > > Key: METRON-1472 > URL: https://issues.apache.org/jira/browse/METRON-1472 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > Add stylelint support for better code quality -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1298) TimeRange Picker doesn't work on Safari
[ https://issues.apache.org/jira/browse/METRON-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495437#comment-16495437 ] ASF GitHub Bot commented on METRON-1298: Github user pono closed the pull request at: https://github.com/apache/metron/pull/830 > TimeRange Picker doesn't work on Safari > --- > > Key: METRON-1298 > URL: https://issues.apache.org/jira/browse/METRON-1298 > Project: Metron > Issue Type: Bug >Reporter: RaghuMitra >Assignee: RaghuMitra >Priority: Major > > Date Picker doesn't work on safari the request goes with a timestamp of NaN -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1474) Add normalizecss
[ https://issues.apache.org/jira/browse/METRON-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495432#comment-16495432 ] ASF GitHub Bot commented on METRON-1474: Github user pono closed the pull request at: https://github.com/apache/metron/pull/956 > Add normalizecss > > > Key: METRON-1474 > URL: https://issues.apache.org/jira/browse/METRON-1474 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Priority: Minor > > Add normalizecss to reduce css mistakes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #830: METRON-1298: TimeRange Picker doesn't work on Safa...
Github user pono closed the pull request at: https://github.com/apache/metron/pull/830 ---
[GitHub] metron pull request #952: METRON-1480 Add yarn as default build tool for the...
Github user pono closed the pull request at: https://github.com/apache/metron/pull/952 ---
[GitHub] metron pull request #954: METRON-1478 Move type definitions to dev-dependenc...
Github user pono closed the pull request at: https://github.com/apache/metron/pull/954 ---
[GitHub] metron pull request #953: METRON-1472 Add stylelint support
Github user pono closed the pull request at: https://github.com/apache/metron/pull/953 ---
[GitHub] metron pull request #951: METRON-1479 Add editorconfig - create guideline fo...
Github user pono closed the pull request at: https://github.com/apache/metron/pull/951 ---
[jira] [Commented] (METRON-1420) Update utilities and components for supporting Solr integration testing
[ https://issues.apache.org/jira/browse/METRON-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495283#comment-16495283 ] Justin Leet commented on METRON-1420: - Some of this already existed and mostly got enhanced as part of other tickets. > Update utilities and components for supporting Solr integration testing > --- > > Key: METRON-1420 > URL: https://issues.apache.org/jira/browse/METRON-1420 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Ryan Merriman >Priority: Major > > Make sure we have any updates to the InMemoryComponent as well as utilities > for creating indices/schemas and loading data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1441) Create complementary Solr schemas for the main sensors
[ https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1441: --- Assignee: Casey Stella > Create complementary Solr schemas for the main sensors > -- > > Key: METRON-1441 > URL: https://issues.apache.org/jira/browse/METRON-1441 > Project: Metron > Issue Type: Sub-task >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > > We have ES templates for bro, snort, yaf, and error, we need corresponding > solr schemas for these collections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1441) Create complementary Solr schemas for the main sensors
[ https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1441: Issue Type: Sub-task (was: Improvement) Parent: METRON-1416 > Create complementary Solr schemas for the main sensors > -- > > Key: METRON-1441 > URL: https://issues.apache.org/jira/browse/METRON-1441 > Project: Metron > Issue Type: Sub-task >Reporter: Casey Stella >Priority: Major > > We have ES templates for bro, snort, yaf, and error, we need corresponding > solr schemas for these collections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1424) Kerberos: Solr
[ https://issues.apache.org/jira/browse/METRON-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1424: --- Assignee: Ryan Merriman > Kerberos: Solr > -- > > Key: METRON-1424 > URL: https://issues.apache.org/jira/browse/METRON-1424 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Ryan Merriman >Priority: Major > > Verify that our Mpack passes the right credentials to Solr. Verify that Solr > kerberizes out of the box. This'll need to be spun up so we can find any > issues. > This should also be spun up for at least the ticket expiration time (over 24 > hours by default). We'd seen some issues with lowering the ticket expiration > time not seeming to be reflected in configs, so we should be careful to make > sure ticket expiration works as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1569) Allow user to change field name conversion when indexing to Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495275#comment-16495275 ] ASF GitHub Bot commented on METRON-1569: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1022 > Allow user to change field name conversion when indexing to Elasticsearch > - > > Key: METRON-1569 > URL: https://issues.apache.org/jira/browse/METRON-1569 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > The `ElasticsearchWriter` has a mechanism to transform the field names of a > message before it is written to Elasticsearch. Right now this mechanism is > hard-coded to replace all '.' dots with ':' colons. > This mechanism was needed for Elasticsearch 2.x which did not allow dots in > field names. Now that Metron supports Elasticsearch 5.x this is no longer a > problem. > A user should be able to configure the field name transformation when writing > to Elasticsearch, as needed. > While it might have been simpler to just remove the de-dotting mechanism, > this would break backwards compatibility. Providing users with a means to > configure this mechanism provides them with an upgrade path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1022: METRON-1569 Allow user to change field name conve...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1022 ---
[GitHub] metron issue #1022: METRON-1569 Allow user to change field name conversion w...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1022 +1 by inspection. This is great @nickwallen, thanks for the contribution. ---
[jira] [Commented] (METRON-1569) Allow user to change field name conversion when indexing to Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495266#comment-16495266 ] ASF GitHub Bot commented on METRON-1569: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1022 +1 by inspection. This is great @nickwallen, thanks for the contribution. > Allow user to change field name conversion when indexing to Elasticsearch > - > > Key: METRON-1569 > URL: https://issues.apache.org/jira/browse/METRON-1569 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > The `ElasticsearchWriter` has a mechanism to transform the field names of a > message before it is written to Elasticsearch. Right now this mechanism is > hard-coded to replace all '.' dots with ':' colons. > This mechanism was needed for Elasticsearch 2.x which did not allow dots in > field names. Now that Metron supports Elasticsearch 5.x this is no longer a > problem. > A user should be able to configure the field name transformation when writing > to Elasticsearch, as needed. > While it might have been simpler to just remove the de-dotting mechanism, > this would break backwards compatibility. Providing users with a means to > configure this mechanism provides them with an upgrade path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1547) Solr Comment Fields
[
https://issues.apache.org/jira/browse/METRON-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495203#comment-16495203
]
ASF GitHub Bot commented on METRON-1547:
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1037
This can be tested both on via REST API and via the UI, although as noted
above there is UI instability for right now.
It should work for both ES and Solr (although right now Solr requires
`index` to be passed with requests, so make sure to add it in the following
steps. `index` will be the same as `sensorType`).
The examples tests are done with curl, but could also be done in Swagger.
## To setup Solr
```
sudo su -
export METRON_HOME=/usr/metron/0.4.3
cd ${METRON_HOME}/bin/
./install_solr.sh
./create_collection.sh bro
./create_collection.sh yaf
./create_collection.sh snort
./create_collection.sh error
./create_collection.sh metaalert
```
Edit the globe config at ${METRON_HOME}/config/zookeeper/global.json to
have "source.type.field"
, e.g.
"geo.hdfs.file" : "/apps/metron/geo/default/GeoLite2-City.mmdb.gz",
"source.type.field" : "source.type"
${METRON_HOME}/bin/zk_load_configs.sh -z node1:2181 -c GLOBAL -m PUSH -i
${METRON_HOME}/config/zookeeper/
Ensure the new config is found.
${METRON_HOME}/bin/zk_load_configs.sh -z node1:2181 -c GLOBAL -m DUMP
In Ambari:
Indexing -> Random Access Search Engine -> Solr
Restart Metron Indexing, Metron REST, and Metron Alerts UI
## Testing
Make sure to replace the guid and add index as needed throughout
### Get a GUID
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"fields": [
"guid"
],
"from": 0,
"indices": [
"bro"
],
"query": "*:*",
"size": 1
}' 'http://node1:8082/api/v1/search/search'
```
Sample Response
```
{
"total": 2120,
"results": [
{
"id": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"source": {
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412"
},
"score": 1,
"index": "bro_index_2018.05.15.16"
}
],
"facetCounts": null
}
```
### Create a new comment
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"comment": "My Comment",
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"sensorType": "bro",
"timestamp": 1526401584951,
"username": "test_username"
}' 'http://node1:8082/api/v1/update/add/comment'
```
### Call findOne
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"sensorType": "bro"
}' 'http://node1:8082/api/v1/search/findOne'
```
Response should contain a new comments field
```
"comments": [
{
"comment": "My Comment",
"username": "test_username",
"timestamp": 1526401584951
}
]
```
### Add another comment
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"comment": "My Comment 2",
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"sensorType": "bro",
"timestamp": 1526401584955,
"username": "test_username_2"
}' 'http://node1:8082/api/v1/update/add/comment'
```
### Patch the comment with a new field
This ensures the raw form is properly translated during patch operations
(otherwise it can be mangled and not be readable later)
```
curl -u user:password -X PATCH --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"patch": [
{
"op": "add",
"path": "/project",
"value": "metron"
}
],
"sensorType": "bro"
}' 'http://node1:8082/api/v1/update/patch'
```
### Find it again
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"sensorType": "bro"
}' 'http://node1:8082/api/v1/search/findOne'
```
Response should have both comments and the new field
```
"comments": [
{
"comment": "My Comment",
"username": "test_username",
"timestamp": 1526401584951
},
{
"comment": "My Comment 2",
"username": "test_username_2",
"timestamp": 1526401584955
}
[GitHub] metron issue #1037: METRON-1547: Solr Comment Fields
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1037
This can be tested both on via REST API and via the UI, although as noted
above there is UI instability for right now.
It should work for both ES and Solr (although right now Solr requires
`index` to be passed with requests, so make sure to add it in the following
steps. `index` will be the same as `sensorType`).
The examples tests are done with curl, but could also be done in Swagger.
## To setup Solr
```
sudo su -
export METRON_HOME=/usr/metron/0.4.3
cd ${METRON_HOME}/bin/
./install_solr.sh
./create_collection.sh bro
./create_collection.sh yaf
./create_collection.sh snort
./create_collection.sh error
./create_collection.sh metaalert
```
Edit the globe config at ${METRON_HOME}/config/zookeeper/global.json to
have "source.type.field"
, e.g.
"geo.hdfs.file" : "/apps/metron/geo/default/GeoLite2-City.mmdb.gz",
"source.type.field" : "source.type"
${METRON_HOME}/bin/zk_load_configs.sh -z node1:2181 -c GLOBAL -m PUSH -i
${METRON_HOME}/config/zookeeper/
Ensure the new config is found.
${METRON_HOME}/bin/zk_load_configs.sh -z node1:2181 -c GLOBAL -m DUMP
In Ambari:
Indexing -> Random Access Search Engine -> Solr
Restart Metron Indexing, Metron REST, and Metron Alerts UI
## Testing
Make sure to replace the guid and add index as needed throughout
### Get a GUID
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"fields": [
"guid"
],
"from": 0,
"indices": [
"bro"
],
"query": "*:*",
"size": 1
}' 'http://node1:8082/api/v1/search/search'
```
Sample Response
```
{
"total": 2120,
"results": [
{
"id": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"source": {
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412"
},
"score": 1,
"index": "bro_index_2018.05.15.16"
}
],
"facetCounts": null
}
```
### Create a new comment
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"comment": "My Comment",
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"sensorType": "bro",
"timestamp": 1526401584951,
"username": "test_username"
}' 'http://node1:8082/api/v1/update/add/comment'
```
### Call findOne
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"sensorType": "bro"
}' 'http://node1:8082/api/v1/search/findOne'
```
Response should contain a new comments field
```
"comments": [
{
"comment": "My Comment",
"username": "test_username",
"timestamp": 1526401584951
}
]
```
### Add another comment
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"comment": "My Comment 2",
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"sensorType": "bro",
"timestamp": 1526401584955,
"username": "test_username_2"
}' 'http://node1:8082/api/v1/update/add/comment'
```
### Patch the comment with a new field
This ensures the raw form is properly translated during patch operations
(otherwise it can be mangled and not be readable later)
```
curl -u user:password -X PATCH --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"patch": [
{
"op": "add",
"path": "/project",
"value": "metron"
}
],
"sensorType": "bro"
}' 'http://node1:8082/api/v1/update/patch'
```
### Find it again
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
"sensorType": "bro"
}' 'http://node1:8082/api/v1/search/findOne'
```
Response should have both comments and the new field
```
"comments": [
{
"comment": "My Comment",
"username": "test_username",
"timestamp": 1526401584951
},
{
"comment": "My Comment 2",
"username": "test_username_2",
"timestamp": 1526401584955
}
]
```
### Remove comment
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"comment": "My Comment",
"guid": "099042a2-ed3f-46df-8d44-2c42e3adf412",
[jira] [Commented] (METRON-1547) Solr Comment Fields
[ https://issues.apache.org/jira/browse/METRON-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495192#comment-16495192 ] ASF GitHub Bot commented on METRON-1547: GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1037 METRON-1547: Solr Comment Fields ## Contributor Comments Adding support for comments on Alerts in Solr. Nested objects are absurdly painful, so this follows the ES expectation of these not needing to be queryable. Solr doesn't have a great way to store these as-is, so they're stored as raw JSON, which is converted appropriately to ensure everything sees the same data as in the ES case. The REST API has been adjusted to have add/remove comment calls, rather than just patching (since that won't work in Solr directly). However, index is required to avoid an error for Solr right now as a result of METRON-1585. Once that is fixed, index will be optional. There are currently a couple bugs that I would love some help with. - In ES, when you add a comment, it doesn't immediately show up on the comments list. It looks like a findOne is fired off, but I assume ES isn't updated yet. This doesn't appear to happen in Solr. - When I hit delete on a comment in the UI (at least in Solr, unsure about ES), the comment is visually removed, but no remove request is fired off. I assume I have something misaligned, but I'm not sure what. - I patterned the optional index off of findOne, because it handles it a bit specially. Delightfully, now I have an extra indexString field showing up in both findOne and the add remove comment. There's some JsonGetter annotation thing I'm probably not using correctly. To test, This can be tested both via REST and the UI. As noted above, at least right now the UI has more instability associated with it. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron solrCommentsBlob Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1037.patch To
[GitHub] metron pull request #1037: METRON-1547: Solr Comment Fields
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1037 METRON-1547: Solr Comment Fields ## Contributor Comments Adding support for comments on Alerts in Solr. Nested objects are absurdly painful, so this follows the ES expectation of these not needing to be queryable. Solr doesn't have a great way to store these as-is, so they're stored as raw JSON, which is converted appropriately to ensure everything sees the same data as in the ES case. The REST API has been adjusted to have add/remove comment calls, rather than just patching (since that won't work in Solr directly). However, index is required to avoid an error for Solr right now as a result of METRON-1585. Once that is fixed, index will be optional. There are currently a couple bugs that I would love some help with. - In ES, when you add a comment, it doesn't immediately show up on the comments list. It looks like a findOne is fired off, but I assume ES isn't updated yet. This doesn't appear to happen in Solr. - When I hit delete on a comment in the UI (at least in Solr, unsure about ES), the comment is visually removed, but no remove request is fired off. I assume I have something misaligned, but I'm not sure what. - I patterned the optional index off of findOne, because it handles it a bit specially. Delightfully, now I have an extra indexString field showing up in both findOne and the add remove comment. There's some JsonGetter annotation thing I'm probably not using correctly. To test, This can be tested both via REST and the UI. As noted above, at least right now the UI has more instability associated with it. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron solrCommentsBlob Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1037.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1037 commit bc44653102f9f628ae23fee1309048dbc07ca0b8 Author: justinjleet Date:
[jira] [Created] (METRON-1585) SolrRetrieveLatestDao does not use the collection lookup
Justin Leet created METRON-1585: --- Summary: SolrRetrieveLatestDao does not use the collection lookup Key: METRON-1585 URL: https://issues.apache.org/jira/browse/METRON-1585 Project: Metron Issue Type: Sub-task Reporter: Justin Leet `getLatest` interface has the second arg as "sensorType", but the Solr DAO makes the assumption that it's "collection" and renames the arg and uses it without retrieving the actual collection. https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrRetrieveLatestDao.java#L47 `getAllLatest` at https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrRetrieveLatestDao.java#L47 This can affect other DAOs that defer to this DAO (e.g. update and metaalert) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1577) Solr searches don't include the index of the result
[
https://issues.apache.org/jira/browse/METRON-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495144#comment-16495144
]
ASF GitHub Bot commented on METRON-1577:
Github user merrimanr closed the pull request at:
https://github.com/apache/metron/pull/1031
> Solr searches don't include the index of the result
> ---
>
> Key: METRON-1577
> URL: https://issues.apache.org/jira/browse/METRON-1577
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> For example
> {code:java}
> {
> "total": 370,
> "results": [
> {
> "id": "1dcf6e7e-9d16-477b-990e-e734bd400101",
> "source":
> {
> "guid": "1dcf6e7e-9d16-477b-990e-e734bd400101"
> },
> "score": 0,
> "index": null
> }
> ],
> "facetCounts": null
> }{code}
> We should also make sure that any other endpoints (if there are any) that
> return index, are populated properly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1031: METRON-1577: Solr searches don't include the inde...
Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/1031 ---
[jira] [Comment Edited] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494991#comment-16494991
]
manisha tank edited comment on METRON-1583 at 5/30/18 12:31 PM:
!cisco_log_error.png!
error message
java.lang.RuntimeException: [Metron] Message 'Oct 24 20:55:00 192.168.10.2
<166>Oct 24 2017 20:53:48: %ASA-6-106100: access-list acl_out denied icmp
outside/192.168.10.2(3) -> inside/172.20.4.75(1) hit-cnt 33 300-second interval
[0x71761f18, 0x0]' does not match pattern '%\{CISCO_TAGGED_SYSLOG}' at
org.apache.metron.parsers.asa.BasicAsaParser.parse(BasicAsaParser.java:184) at
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45)
at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177) at
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
at
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
at
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: [Metron] Message 'Oct 24 20:55:00
192.168.10.2 <166>Oct 24 2017 20:53:48: %ASA-6-106100: access-list acl_out
denied icmp outside/192.168.10.2(3) -> inside/172.20.4.75(1) hit-cnt 33
300-second interval [0x71761f18, 0x0]' does not match pattern
'%\{CISCO_TAGGED_SYSLOG}' at
org.apache.metron.parsers.asa.BasicAsaParser.parse(BasicAsaParser.java:178) ...
12 more
was (Author: manisha):
!cisco_log_error.png!
error message
java.lang.RuntimeException: [Metron] Message 'Oct 24 20:55:00 192.168.10.2
<166>Oct 24 2017 20:53:48: %ASA-6-106100: access-list acl_out denied icmp
outside/192.168.10.2(3) -> inside/172.20.4.75(1) hit-cnt 33 300-second interval
[0x71761f18, 0x0]' does not match pattern '%\{CISCO_TAGGED_SYSLOG}' at
org.apache.metron.parsers.asa.BasicAsaParser.parse(BasicAsaParser.java:184) at
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45)
at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177) at
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
at
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
at
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: [Metron] Message 'Oct 24 20:55:00
192.168.10.2 <166>Oct 24 2017 20:53:48: %ASA-6-106100: access-list acl_out
denied icmp outside/192.168.10.2(3) -> inside/172.20.4.75(1) hit-cnt 33
300-second interval [0x71761f18, 0x0]' does not match pattern
'%\{CISCO_TAGGED_SYSLOG}' at
org.apache.metron.parsers.asa.BasicAsaParser.parse(BasicAsaParser.java:178) ...
12 more
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_log_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25
[jira] [Comment Edited] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494991#comment-16494991
]
manisha tank edited comment on METRON-1583 at 5/30/18 12:31 PM:
!cisco_log_error.png!
error message
java.lang.RuntimeException: [Metron] Message 'Oct 24 20:55:00 192.168.10.2
<166>Oct 24 2017 20:53:48: %ASA-6-106100: access-list acl_out denied icmp
outside/192.168.10.2(3) -> inside/172.20.4.75(1) hit-cnt 33 300-second interval
[0x71761f18, 0x0]' does not match pattern '%\{CISCO_TAGGED_SYSLOG}' at
org.apache.metron.parsers.asa.BasicAsaParser.parse(BasicAsaParser.java:184) at
org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45)
at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:177) at
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
at
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
at
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: [Metron] Message 'Oct 24 20:55:00
192.168.10.2 <166>Oct 24 2017 20:53:48: %ASA-6-106100: access-list acl_out
denied icmp outside/192.168.10.2(3) -> inside/172.20.4.75(1) hit-cnt 33
300-second interval [0x71761f18, 0x0]' does not match pattern
'%\{CISCO_TAGGED_SYSLOG}' at
org.apache.metron.parsers.asa.BasicAsaParser.parse(BasicAsaParser.java:178) ...
12 more
was (Author: manisha):
!cisco_log_error.png!
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_log_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494991#comment-16494991
]
manisha tank edited comment on METRON-1583 at 5/30/18 10:44 AM:
!cisco_log_error.png!
was (Author: manisha):
Yes Sure
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_log_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
manisha tank updated METRON-1583:
-
Attachment: cisco_log_error.png
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_log_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
manisha tank updated METRON-1583:
-
Attachment: (was: cisco_log_error.png)
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_log_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
manisha tank updated METRON-1583:
-
Attachment: (was: cisco_asa_logs_error.png)
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_log_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494991#comment-16494991
]
manisha tank commented on METRON-1583:
--
Yes Sure
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_log_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
manisha tank updated METRON-1583:
-
Attachment: cisco_log_error.png
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_asa_logs_error.png, cisco_log_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
