[GitHub] metron issue #754: METRON-1184 EC2 Deployment - Updating control_path to acc...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/754 FYI, I just attempted to spin up an EC2 instance and it failed for me. Switching back to the previous control_path setting fixed it. ---
[jira] [Commented] (METRON-1184) EC2 Deployment - Updating control_path to accommodate for Linux
[
https://issues.apache.org/jira/browse/METRON-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16486522#comment-16486522
]
ASF GitHub Bot commented on METRON-1184:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/754
FYI, I just attempted to spin up an EC2 instance and it failed for me.
Switching back to the previous control_path setting fixed it.
> EC2 Deployment - Updating control_path to accommodate for Linux
> ---
>
> Key: METRON-1184
> URL: https://issues.apache.org/jira/browse/METRON-1184
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 0.4.1
> Environment: ClearOS[Distro of CentOS] npm 3.10.10, node 6.11.1,
> docker 1.12.6, mvn 3.3.9, ansible 2.2.2.0, No Vagrant)
>Reporter: ashah
>Assignee: ashah
>Priority: Minor
> Labels: amazon, cloud
> Fix For: 0.5.0
>
>
> PR: https://github.com/apache/metron/pull/754
> For 0.4.1-rc4, using "control_path = ~/.ssh/ansible-ssh-%%C" for a Linux
> environment ( such as ClearOS[Distro of CentOS] npm 3.10.10, node 6.11.1,
> docker 1.12.6, mvn 3.3.9, ansible 2.2.2.0, No Vagrant) could throw the
> following error:
> ```
> "TASK [setup]
> ***
> fatal: [ec2-xx-yy-0-130.us-west-2.compute.amazonaws.com]: UNREACHABLE! =>
> {"changed": false, "msg": "Failed to connect to the host via ssh:
> percent_expand: unknown key %C\r\n", "unreachable": true}"
> ```
> This PR is to change the line to "control_path = ~/.ssh/ansbile-ssh-%%h-%%r"
> to make amazon-ec2 also potentially deployable with Linux while preserving a
> small path length.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1574) Update version to 0.5.0
[ https://issues.apache.org/jira/browse/METRON-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1574: Fix Version/s: 0.5.0 > Update version to 0.5.0 > --- > > Key: METRON-1574 > URL: https://issues.apache.org/jira/browse/METRON-1574 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.5.0 > > > For the next release, we decided to go with 0.5.0, rather than 0.4.3. This > requires bumping the version number per > https://cwiki.apache.org/confluence/display/METRON/Release+Process and > https://cwiki.apache.org/confluence/display/METRON/Change+the+Build+Version+Number -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1574) Update version to 0.5.0
[ https://issues.apache.org/jira/browse/METRON-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16486499#comment-16486499 ] ASF GitHub Bot commented on METRON-1574: GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1026 METRON-1574: Update version to 0.5.0 ## Contributor Comments Bumping the version number per instructions at https://cwiki.apache.org/confluence/display/METRON/Change+the+Build+Version+Number. Let me know if there's either anything I missed changing or anything I shouldn't have changed. I did a quick smoke test and ran it up in full dev to make sure data was flowing through and visible in the UI, but we can do more extensive testing if we want. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron versionBump Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1026.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1026 commit c71d526c10579416c81ed61ed7bb258394c49ad8 Author: justinjleet Date: 2018-05-23T00:06:28Z bumping version to 0.5.0 from 0.4.3 > Update version to 0.5.0 > --- > > Key: METRON-1574 > URL: https://issues.apache.org/jira/browse/METRON-1574 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > For the next release, we decided to go with 0.5.0, rather than 0.4.3. This > requires bumping the version number per > https://cwiki.apache.org/confluence/display/METRON/Release+Process and > https://cwiki.apache.org/confluence/display/METRON/Change+the+Build+Version+Number -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1026: METRON-1574: Update version to 0.5.0
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1026 METRON-1574: Update version to 0.5.0 ## Contributor Comments Bumping the version number per instructions at https://cwiki.apache.org/confluence/display/METRON/Change+the+Build+Version+Number. Let me know if there's either anything I missed changing or anything I shouldn't have changed. I did a quick smoke test and ran it up in full dev to make sure data was flowing through and visible in the UI, but we can do more extensive testing if we want. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron versionBump Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1026.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1026 commit c71d526c10579416c81ed61ed7bb258394c49ad8 Author: justinjleet Date: 2018-05-23T00:06:28Z bumping version to 0.5.0 from 0.4.3 ---
[jira] [Created] (METRON-1574) Update version to 0.5.0
Justin Leet created METRON-1574: --- Summary: Update version to 0.5.0 Key: METRON-1574 URL: https://issues.apache.org/jira/browse/METRON-1574 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet For the next release, we decided to go with 0.5.0, rather than 0.4.3. This requires bumping the version number per https://cwiki.apache.org/confluence/display/METRON/Release+Process and https://cwiki.apache.org/confluence/display/METRON/Change+the+Build+Version+Number -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1573) Enhance KAFKA_* functions to return partition and offset details
Nick Allen created METRON-1573: -- Summary: Enhance KAFKA_* functions to return partition and offset details Key: METRON-1573 URL: https://issues.apache.org/jira/browse/METRON-1573 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen The KAFKA_* functions currently simply return the value of the message. There are often times when it would be useful to get more detailed information including the message partition, offset, key, etc. The functions should be enhanced to allow a user to optionally get a more detailed view. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1000: METRON-1533 Create KAFKA_FIND Stellar Function
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1000 I made a bunch of enhancements based on the feedback I outlined above. I am in the process of breaking that work out into multiple PRs so that it can be reviewed more easily. ---
[GitHub] metron pull request #1000: METRON-1533 Create KAFKA_FIND Stellar Function
Github user nickwallen closed the pull request at: https://github.com/apache/metron/pull/1000 ---
[jira] [Commented] (METRON-1533) Create KAFKA_FIND Stellar Function
[
https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16485762#comment-16485762
]
ASF GitHub Bot commented on METRON-1533:
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1000
I made a bunch of enhancements based on the feedback I outlined above. I
am in the process of breaking that work out into multiple PRs so that it can be
reviewed more easily.
> Create KAFKA_FIND Stellar Function
> --
>
> Key: METRON-1533
> URL: https://issues.apache.org/jira/browse/METRON-1533
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> When creating enrichments, I often find that I want to validate that the
> enrichment I just created was successful on the live, incoming stream of
> telemetry. My workflow looks something like this.
> 1. Create and test the enrichment that I want to create.
> {code:java}
> [Stellar]>>> ip_src_addr := "72.34.49.86"
> 72.34.49.86
> [Stellar]>>> geo := GEO_GET(ip_src_addr)
> {country=US, dmaCode=803, city=Los Angeles, postalCode=90014,
> latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361,
> longitude=-118.2512}
> {code}
> 2. That looks good to me. Now let's add that to my Bro telemetry.
> {code:java}
> [Stellar]>>> conf := SHELL_EDIT(conf)
> {
> "enrichment" : {
> "fieldMap": {
> "stellar": {
> "config": [
>"geo := GEO_GET(ip_src_addr)"
> ]
> }
> }
> },
> "threatIntel": {
> }
> }
> [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
> {code}
>
> 3. It looks like that worked, but did that really work?
> At this point, I would run KAFKA_GET as many times as it takes to retrieve a
> Bro message. You would just have to get lucky and hope that the enrichment
> worked and secondly that you would pull down a Bro message (as opposed to a
> different sensor).
>
> I would rather have a function that lets me only pull back the messages that
> I care about. In this case I could either retrieve only Bro messages.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
> {code}
> Or I could look for messages that contain geolocation data.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1533) Create KAFKA_FIND Stellar Function
[
https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16485763#comment-16485763
]
ASF GitHub Bot commented on METRON-1533:
Github user nickwallen closed the pull request at:
https://github.com/apache/metron/pull/1000
> Create KAFKA_FIND Stellar Function
> --
>
> Key: METRON-1533
> URL: https://issues.apache.org/jira/browse/METRON-1533
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> When creating enrichments, I often find that I want to validate that the
> enrichment I just created was successful on the live, incoming stream of
> telemetry. My workflow looks something like this.
> 1. Create and test the enrichment that I want to create.
> {code:java}
> [Stellar]>>> ip_src_addr := "72.34.49.86"
> 72.34.49.86
> [Stellar]>>> geo := GEO_GET(ip_src_addr)
> {country=US, dmaCode=803, city=Los Angeles, postalCode=90014,
> latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361,
> longitude=-118.2512}
> {code}
> 2. That looks good to me. Now let's add that to my Bro telemetry.
> {code:java}
> [Stellar]>>> conf := SHELL_EDIT(conf)
> {
> "enrichment" : {
> "fieldMap": {
> "stellar": {
> "config": [
>"geo := GEO_GET(ip_src_addr)"
> ]
> }
> }
> },
> "threatIntel": {
> }
> }
> [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
> {code}
>
> 3. It looks like that worked, but did that really work?
> At this point, I would run KAFKA_GET as many times as it takes to retrieve a
> Bro message. You would just have to get lucky and hope that the enrichment
> worked and secondly that you would pull down a Bro message (as opposed to a
> different sensor).
>
> I would rather have a function that lets me only pull back the messages that
> I care about. In this case I could either retrieve only Bro messages.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
> {code}
> Or I could look for messages that contain geolocation data.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1533) Create KAFKA_FIND Stellar Function
[
https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16485759#comment-16485759
]
ASF GitHub Bot commented on METRON-1533:
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1025
METRON-1533 Create KAFKA_FIND Stellar function
This PR is built on #1024 and #1023. Dig into the last commit to review
the changes for this PR alone.
### Changes
I created a `KAFKA_FIND` function that allows you to provide a filter
expression so that only messages satisfying a condition are returned. For
example...
- Find a message that has been enriched with geolocation data.
```
KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
```
- Find a Bro message.
```
KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
```
The message is presented to the filter lambda expression as a map of field
values. This makes creating the filter expression a bit simpler.
Like the other `KAFKA_*` functions, this is not intended to be highly
performant. This is only intended to make the process of creating and
modifying enrichments simpler in the REPL. See the **Use Case** section for
more details on how I see this being used.
### Future
If we were in the future to provide map literals in Stellar, this would
become a fair bit simpler.
```
KAFKA_FIND('indexing', m -> m['source.type'] == 'bro')
```
### Use Case
When creating enrichments, I often find that I want to validate that the
enrichment I just created was successful on the live, incoming stream of
telemetry. My workflow looks something like this.
1. Create and test the enrichment that I want to create.
```
[Stellar]>>> ip_src_addr := "72.34.49.86"
72.34.49.86
[Stellar]>>> geo := GEO_GET(ip_src_addr)
{country=US, dmaCode=803, city=Los Angeles, postalCode=90014,
latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361,
longitude=-118.2512}
```
2. That looks good to me. Now let's add that to my Bro telemetry.
```
[Stellar]>>> conf := SHELL_EDIT(conf)
{
"enrichment" : {
"fieldMap": {
"stellar": {
"config": [
"geo := GEO_GET(ip_src_addr)"
]
}
}
},
"threatIntel": {
}
}
[Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
```
3. It looks like that worked, but did that really work?
At this point, I would run `KAFKA_GET` as many times as it takes to
retrieve a Bro message. You would just have to get lucky and hope that the
enrichment worked and secondly that you would pull down a Bro message (as
opposed to a different sensor).
I would rather have a function that lets me only pull back the messages
that I care about. In this case I could either retrieve only Bro messages.
```
KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
```
Or I could look for messages that contain geolocation data.
```
KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
```
### Changes
* Created the `KAFKA_FIND` function along with unit tests.
* Updated all KAFKA_* functions to use a standard `getArg` function so that
argument handling is all done the same way.
### Pull Request Checklist
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge
[GitHub] metron pull request #1025: METRON-1533 Create KAFKA_FIND Stellar function
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1025
METRON-1533 Create KAFKA_FIND Stellar function
This PR is built on #1024 and #1023. Dig into the last commit to review
the changes for this PR alone.
### Changes
I created a `KAFKA_FIND` function that allows you to provide a filter
expression so that only messages satisfying a condition are returned. For
example...
- Find a message that has been enriched with geolocation data.
```
KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
```
- Find a Bro message.
```
KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
```
The message is presented to the filter lambda expression as a map of field
values. This makes creating the filter expression a bit simpler.
Like the other `KAFKA_*` functions, this is not intended to be highly
performant. This is only intended to make the process of creating and
modifying enrichments simpler in the REPL. See the **Use Case** section for
more details on how I see this being used.
### Future
If we were in the future to provide map literals in Stellar, this would
become a fair bit simpler.
```
KAFKA_FIND('indexing', m -> m['source.type'] == 'bro')
```
### Use Case
When creating enrichments, I often find that I want to validate that the
enrichment I just created was successful on the live, incoming stream of
telemetry. My workflow looks something like this.
1. Create and test the enrichment that I want to create.
```
[Stellar]>>> ip_src_addr := "72.34.49.86"
72.34.49.86
[Stellar]>>> geo := GEO_GET(ip_src_addr)
{country=US, dmaCode=803, city=Los Angeles, postalCode=90014,
latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361,
longitude=-118.2512}
```
2. That looks good to me. Now let's add that to my Bro telemetry.
```
[Stellar]>>> conf := SHELL_EDIT(conf)
{
"enrichment" : {
"fieldMap": {
"stellar": {
"config": [
"geo := GEO_GET(ip_src_addr)"
]
}
}
},
"threatIntel": {
}
}
[Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
```
3. It looks like that worked, but did that really work?
At this point, I would run `KAFKA_GET` as many times as it takes to
retrieve a Bro message. You would just have to get lucky and hope that the
enrichment worked and secondly that you would pull down a Bro message (as
opposed to a different sensor).
I would rather have a function that lets me only pull back the messages
that I care about. In this case I could either retrieve only Bro messages.
```
KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
```
Or I could look for messages that contain geolocation data.
```
KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
```
### Changes
* Created the `KAFKA_FIND` function along with unit tests.
* Updated all KAFKA_* functions to use a standard `getArg` function so that
argument handling is all done the same way.
### Pull Request Checklist
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1533-NEW
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1025.patch
To close th
[jira] [Commented] (METRON-1572) Enhance KAFKA_PUT function
[ https://issues.apache.org/jira/browse/METRON-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16485742#comment-16485742 ] ASF GitHub Bot commented on METRON-1572: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1024 METRON-1572 Enhance KAFKA_PUT function Note: This PR is built on the shoulders of #1023. To review the change specific to this PR, review the changes made in the second commit or wait until #1023 is merged. ### Changes * Enhances the KAFKA_PUT function to accept either a list of strings or a string. This makes it simpler to send either 1 message or a bunch of messages. * KAFKA_PUT should queue up all messages to be sent before waiting on a response. This improves response time when sending a large number of messages. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1572 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1024.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1024 commit 10fe2bfd04db6633b73e9d0f37d458f4ce83aaf9 Author: Nick Allen Date: 2018-04-20T20:18:16Z METRON-1571 commit 75e9f39fe55c07cd149f00e26c0ec1de760e2d9a Author: Nick Allen Date: 2018-05-22T23:17:09Z METRON-1572 Enhance KAFKA_PUT function > Enhance KAFKA_PUT function > -- > > Key: METRON-1572 > URL: https://issues.apache.org/jira/browse/METRON-1572 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > Enhance the KAFKA_PUT function to accept either a List of String or a String. > This makes it simpler to send either 1 message or a bunch of messages. > KAFKA_PUT should queue up all messages to be sent before waiting on a > response. This improves response when sending a large number of messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1024: METRON-1572 Enhance KAFKA_PUT function
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1024 METRON-1572 Enhance KAFKA_PUT function Note: This PR is built on the shoulders of #1023. To review the change specific to this PR, review the changes made in the second commit or wait until #1023 is merged. ### Changes * Enhances the KAFKA_PUT function to accept either a list of strings or a string. This makes it simpler to send either 1 message or a bunch of messages. * KAFKA_PUT should queue up all messages to be sent before waiting on a response. This improves response time when sending a large number of messages. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1572 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1024.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1024 commit 10fe2bfd04db6633b73e9d0f37d458f4ce83aaf9 Author: Nick Allen Date: 2018-04-20T20:18:16Z METRON-1571 commit 75e9f39fe55c07cd149f00e26c0ec1de760e2d9a Author: Nick Allen Date: 2018-05-22T23:17:09Z METRON-1572 Enhance KAFKA_PUT function ---
[jira] [Created] (METRON-1572) Enhance KAFKA_PUT function
Nick Allen created METRON-1572: -- Summary: Enhance KAFKA_PUT function Key: METRON-1572 URL: https://issues.apache.org/jira/browse/METRON-1572 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen Enhance the KAFKA_PUT function to accept either a List of String or a String. This makes it simpler to send either 1 message or a bunch of messages. KAFKA_PUT should queue up all messages to be sent before waiting on a response. This improves response when sending a large number of messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1571) Correct KAFKA_TAIL Seek to End Logic
[ https://issues.apache.org/jira/browse/METRON-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484726#comment-16484726 ] ASF GitHub Bot commented on METRON-1571: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1023 METRON-1571 Correct KAFKA_TAIL Seek to End Logic ## Changes * KAFKA_TAIL now performs manual partition assignment and correctly uses the Kafka API to seek to the end of the topic. * Previously, the function messed with the 'group.id' to effectively seek to end. This is an abuse of the API. Also, a fixed group.id would be required for using the KAFKA_* functions in a Kerberized cluster. * KAKFA_GET and KAFKA_TAIL now use a more accurate mechanism for adhering to the user's requested max wait time. * Previously, the max wait was divided by the max poll timeout to estimate how many times Kafka should be polled before returning to the user. This method is not always accurate when some poll requests return values and others do not. * The MPack now defines a global property `bootstrap.servers` which allow the KAFKA_* functions to work out-of-the-box after a Metron installation. * Previously, a user had to manually define this property before using the KAKFA_* functions. If the property is not defined correctly, they would have to wait an excessive amount of time for the connection to timeout since there is no means in the REPL to interrupt a function call. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1571 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1023.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1023 commit 10fe2bfd04db6633b73e9d0f37d458f4ce83aaf9 Author: Nick Allen Date: 2018-04-20T20:18:16Z METRON-1571 > Correct KAFKA_TAIL Seek to End Logic > > > Key: METRON-1571 > URL: https://issues.apache.org/jira/browse/METRON-1571 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Priority: Major > > KAFKA_TAIL always needs to tail from the end of a topic. The current > implementation does not correctly seek to the end of the topic as the Kafka > API supports. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1023: METRON-1571 Correct KAFKA_TAIL Seek to End Logic
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1023 METRON-1571 Correct KAFKA_TAIL Seek to End Logic ## Changes * KAFKA_TAIL now performs manual partition assignment and correctly uses the Kafka API to seek to the end of the topic. * Previously, the function messed with the 'group.id' to effectively seek to end. This is an abuse of the API. Also, a fixed group.id would be required for using the KAFKA_* functions in a Kerberized cluster. * KAKFA_GET and KAFKA_TAIL now use a more accurate mechanism for adhering to the user's requested max wait time. * Previously, the max wait was divided by the max poll timeout to estimate how many times Kafka should be polled before returning to the user. This method is not always accurate when some poll requests return values and others do not. * The MPack now defines a global property `bootstrap.servers` which allow the KAFKA_* functions to work out-of-the-box after a Metron installation. * Previously, a user had to manually define this property before using the KAKFA_* functions. If the property is not defined correctly, they would have to wait an excessive amount of time for the connection to timeout since there is no means in the REPL to interrupt a function call. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1571 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1023.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1023 commit 10fe2bfd04db6633b73e9d0f37d458f4ce83aaf9 Author: Nick Allen Date: 2018-04-20T20:18:16Z METRON-1571 ---
[jira] [Created] (METRON-1571) Correct KAFKA_TAIL Seek to End Logic
Nick Allen created METRON-1571: -- Summary: Correct KAFKA_TAIL Seek to End Logic Key: METRON-1571 URL: https://issues.apache.org/jira/browse/METRON-1571 Project: Metron Issue Type: Improvement Reporter: Nick Allen KAFKA_TAIL always needs to tail from the end of a topic. The current implementation does not correctly seek to the end of the topic as the Kafka API supports. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1570) Downgrade Solr version to match HDP Search
Ryan Merriman created METRON-1570: - Summary: Downgrade Solr version to match HDP Search Key: METRON-1570 URL: https://issues.apache.org/jira/browse/METRON-1570 Project: Metron Issue Type: Bug Reporter: Ryan Merriman We should be building and testing with the same version as HDP Search (5.5.4). We need to bump our Maven version down to 5.5.4, make any necessary adjustments in our Solr code and integration/unit tests, and verify everything still works in full dev including Kerberos. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1462) Separate ES and Kibana from Metron Mpack
[ https://issues.apache.org/jira/browse/METRON-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1462: Fix Version/s: 0.5.0 > Separate ES and Kibana from Metron Mpack > > > Key: METRON-1462 > URL: https://issues.apache.org/jira/browse/METRON-1462 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.5.0 > > > Need this to enable our ability to provide pluggable Solr support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1468) Add support for apache/metron-bro-plugin-kafka to prepare-commit
[ https://issues.apache.org/jira/browse/METRON-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1468: Fix Version/s: 0.5.0 > Add support for apache/metron-bro-plugin-kafka to prepare-commit > > > Key: METRON-1468 > URL: https://issues.apache.org/jira/browse/METRON-1468 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Fix For: 0.5.0 > > > This is to add support for apache/metron-bro-plugin-kafka to prepare-commit. > It should still default to apache/metron across the board, but allows you to > specify metron-bro-plugin-kafka when prompted, which should propagate > properly throughout the rest of the script. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1520) Add caching for stellar field transformations
[ https://issues.apache.org/jira/browse/METRON-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1520: Fix Version/s: 0.5.0 > Add caching for stellar field transformations > - > > Key: METRON-1520 > URL: https://issues.apache.org/jira/browse/METRON-1520 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Given how important caching is in the enrichment topology, we should have > caching for stellar field transformations in the parsers as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1429) SearchIntegrationTest refactor
[ https://issues.apache.org/jira/browse/METRON-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1429: Fix Version/s: 0.5.0 > SearchIntegrationTest refactor > -- > > Key: METRON-1429 > URL: https://issues.apache.org/jira/browse/METRON-1429 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.5.0 > > > In anticipation of METRON-1419, we need to do a slight refactor of > SearchIntegrationTest to make adding a Solr implementation easier. This will > include removing redundant tests and dependencies on meta alerts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1505) Intermittent Profiler Integration Test Failure
[
https://issues.apache.org/jira/browse/METRON-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1505:
Fix Version/s: (was: Next + 1)
0.5.0
> Intermittent Profiler Integration Test Failure
> --
>
> Key: METRON-1505
> URL: https://issues.apache.org/jira/browse/METRON-1505
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.5.0
>
>
> The Profiler integration tests which use processing time fail intermittently
> when run in Travis CI.
> {code:java}
> 2018-03-22 22:00:35 DEBUG FixedFrequencyFlushSignal:67 - Flush counters reset
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 0 route(s) for
> message with timestamp=1521756035759
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 0 route(s) for
> message with timestamp=1521756035802
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 0 route(s) for
> message with timestamp=1521756035806
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message;
> profile=example2, entity=10.0.0.2, timestamp=1521756035807
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
> message with timestamp=1521756035807
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message;
> profile=example2, entity=10.0.0.2, timestamp=1521756035808
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
> message with timestamp=1521756035808
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message;
> profile=example2, entity=10.0.0.2, timestamp=1521756035813
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
> message with timestamp=1521756035813
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message;
> profile=example2, entity=10.0.0.3, timestamp=1521756035814
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
> message with timestamp=1521756035814
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message;
> profile=example2, entity=10.0.0.3, timestamp=1521756035816
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
> message with timestamp=1521756035816
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message;
> profile=example2, entity=10.0.0.3, timestamp=1521756035817
> 2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
> message with timestamp=1521756035817
> 2018-03-22 22:00:41 DEBUG WindowManager:189 - Scan events, eviction policy
> TimeEvictionPolicy{windowLength=5000, referenceTime=1521756041122}
> 2018-03-22 22:00:41 DEBUG WindowManager:212 - [6] events expired from window.
> 2018-03-22 22:00:41 DEBUG WindowManager:214 - invoking
> windowLifecycleListener.onExpiry
> 2018-03-22 22:00:41 DEBUG WindowManager:147 - No events in the window,
> skipping onActivation
> 2018-03-22 22:00:46 DEBUG WindowManager:189 - Scan events, eviction policy
> TimeEvictionPolicy{windowLength=5000, referenceTime=1521756046122}
> 2018-03-22 22:00:46 DEBUG WindowManager:212 - [0] events expired from window.
> 2018-03-22 22:00:46 DEBUG WindowManager:147 - No events in the window,
> skipping onActivation
> 2018-03-22 22:00:51 DEBUG WindowManager:189 - Scan events, eviction policy
> TimeEvictionPolicy{windowLength=5000, referenceTime=1521756051122}
> 2018-03-22 22:00:51 DEBUG WindowManager:212 - [0] events expired from window.
> 2018-03-22 22:00:51 DEBUG WindowManager:147 - No events in the window,
> skipping onActivation
> 2018-03-22 22:00:56 DEBUG WindowManager:189 - Scan events, eviction policy
> TimeEvictionPolicy{windowLength=5000, referenceTime=1521756056122}
> 2018-03-22 22:00:56 DEBUG WindowManager:212 - [0] events expired from window.
> 2018-03-22 22:00:56 DEBUG WindowManager:144 - invoking
> windowLifecycleListener onActivation, [1] events in window.
> 2018-03-22 22:00:56 DEBUG ProfileBuilderBolt:276 - Tuple window contains 1
> tuple(s), 0 expired, 1 new
> 2018-03-22 22:00:56 DEBUG ProfileBuilderBolt:365 - Emitted 0 measurement(s).
> 2018-03-22 22:00:56 DEBUG ProfileBuilderBolt:325 - Flushed expired profiles
> and found 0 measurement(s).
> 2018-03-22 22:00:56 DEBUG FixedFrequencyFlushSignal:114 - Flush=false, '0' ms
> until flush; currentTime=0, flushTime=0
> 2018-03-22 22:01:01 DEBUG WindowManager:189 - Scan events, eviction policy
> TimeEvictionPolicy{windowLength=5000, referenceTime=1521756061122}
> 2018-03-22 22:01:01 DEBUG WindowManager:212 - [1] events expired from window.
> 2018-03-22 22:01:01 DEBUG WindowManager:214 - invoking
> windowLifecycleListener.onExpiry
> 2018-03-22 22:01:01 DEBUG WindowManager:147 - No events in the window,
> skipping onActivation
> 2018-03-22
[jira] [Updated] (METRON-1490) Better error message when user specifies an enrichment type that doesn't exist
[ https://issues.apache.org/jira/browse/METRON-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1490: Fix Version/s: 0.5.0 > Better error message when user specifies an enrichment type that doesn't exist > -- > > Key: METRON-1490 > URL: https://issues.apache.org/jira/browse/METRON-1490 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > If a user specifies an enrichment type that doesn't exist (e.g. > hbaseEnrichment vs hbaseThreatIntel), then we NPE rather than express the > issue in the logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1395) Documentation missing for Produce a message to a Kafka topic Rest API endpoint
[
https://issues.apache.org/jira/browse/METRON-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1395:
Fix Version/s: 0.5.0
> Documentation missing for Produce a message to a Kafka topic Rest API endpoint
> --
>
> Key: METRON-1395
> URL: https://issues.apache.org/jira/browse/METRON-1395
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Minor
> Fix For: 0.5.0
>
>
> Documentation missing for Produces a message to a Kafka topic API
> (/api/v1/kafka/topic/{name}/produce)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1446) Fix openjdk issue with Ubuntu
[ https://issues.apache.org/jira/browse/METRON-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1446: Fix Version/s: 0.5.0 > Fix openjdk issue with Ubuntu > - > > Key: METRON-1446 > URL: https://issues.apache.org/jira/browse/METRON-1446 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.5.0 > > > There appears to have been an update to how openjdk packages are setup with > Ubuntu. The setup requests user feedback that is hanging the ansible command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1463) Adjust the groupings and shuffles in enrichment to be more efficient
[ https://issues.apache.org/jira/browse/METRON-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1463: Fix Version/s: 0.5.0 > Adjust the groupings and shuffles in enrichment to be more efficient > > > Key: METRON-1463 > URL: https://issues.apache.org/jira/browse/METRON-1463 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Currently there are some deficiencies in our grouping approach in the > enrichment topology: > * We have field grouping by key in places where we need LOCAL_OR_SHUFFLE > groupings > * We have shuffle groupings in places where we need LOCAL_OR_SHUFFLE > groupings > * We have field groupings by key in places where we need field grouping by > message (specifically in the connections from the splitter to the > enrichments). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1522) Fix the typo errors at profile debugger readme
[
https://issues.apache.org/jira/browse/METRON-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1522:
Fix Version/s: 0.5.0
> Fix the typo errors at profile debugger readme
> ---
>
> Key: METRON-1522
> URL: https://issues.apache.org/jira/browse/METRON-1522
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Trivial
> Fix For: 0.5.0
>
>
> https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles
>
> There is typo error at step 6 , The profile name has been used as 'p' and
> 'profiler' at places .
> {code:java}
> [Stellar]>>> values := PROFILER_FLUSH(profiler)
> {code}
> it should be
> {code:java}
> [Stellar]>>> values := PROFILER_FLUSH(p)
> {code}
> Or to keep it consistent across the sections the profile definition should be
> renamed to 'profiler' instead of 'p'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1457) Move ASF links to main page in the Metron website
[ https://issues.apache.org/jira/browse/METRON-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1457: Fix Version/s: 0.5.0 > Move ASF links to main page in the Metron website > - > > Key: METRON-1457 > URL: https://issues.apache.org/jira/browse/METRON-1457 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.5.0 > > > The latest changes that went out as a part of METRON-1386 > ([https://github.com/apache/metron/pull/935]) still did not fix the > requisites per: > [https://whimsy.apache.org/site/project/metron] > > Apparently, whimsy looks for the relevant links in the main index.html page > and not sub-links. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1504) Enriching missing values does not match the semantics between the new enrichment topology and old
[ https://issues.apache.org/jira/browse/METRON-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1504: Fix Version/s: 0.5.0 > Enriching missing values does not match the semantics between the new > enrichment topology and old > - > > Key: METRON-1504 > URL: https://issues.apache.org/jira/browse/METRON-1504 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Currently if one tries to enrich an missing value (e.g. via a hbase > enrichment) with the new enrichment topology, you get NPEs. The existing > semantics from the old enrichment topology is for no enrichment to happen. > The new enrichment topology should be made to match this semantic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1545) Upgrade Spring and Spring Boot
[ https://issues.apache.org/jira/browse/METRON-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1545: Fix Version/s: 0.5.0 > Upgrade Spring and Spring Boot > -- > > Key: METRON-1545 > URL: https://issues.apache.org/jira/browse/METRON-1545 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.5.0 > > > The metron-rest module depends on old versions of Spring and Spring Boot. We > should upgrade these to the latest release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1365) Allow PROFILE_GET to return a default value for a profile and entity that does not have a value written.
[ https://issues.apache.org/jira/browse/METRON-1365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1365: Fix Version/s: 0.5.0 > Allow PROFILE_GET to return a default value for a profile and entity that > does not have a value written. > > > Key: METRON-1365 > URL: https://issues.apache.org/jira/browse/METRON-1365 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Right now the profiler is a sparse system, namely if data is not written > during a profile period for a given profile and entity, no values are written > to HBase. Some algorithms may need non-sparse data (e.g. time series outlier > detection algorithms). We should keep the default behavior as it stands, but > allow the user at read time to specify a default value. This is most cleanly > done by using the existing profiler config overrides (the 3rd argument to > PROFILE_GET) and adding a profiler.default.value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1230) As a stopgap prior to METRON-777, add more simplistic sideloading of custom Parsers
[ https://issues.apache.org/jira/browse/METRON-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1230: Fix Version/s: 0.5.0 > As a stopgap prior to METRON-777, add more simplistic sideloading of custom > Parsers > --- > > Key: METRON-1230 > URL: https://issues.apache.org/jira/browse/METRON-1230 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Until we get 777 in, it'd be nice to have a simple ability using the normal > storm functionality to enable users to provide custom parsers without forking > Metron. This should be done via simply creating a jar with their code (and > bundled dependencies) and have it picked up and available to the REST Service > (and consequently the management UI) as well as the start_parser_topology.sh > command. > This should be minimal movement as we have a more robust solution coming with > METRON-777. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-939: --- Fix Version/s: 0.5.0 > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic >Priority: Major > Labels: backwards-incompatible > Fix For: 0.5.0 > > Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - > Kibana.pdf > > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1386) Fix Metron Website Required Links
[ https://issues.apache.org/jira/browse/METRON-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1386: Fix Version/s: 0.5.0 > Fix Metron Website Required Links > - > > Key: METRON-1386 > URL: https://issues.apache.org/jira/browse/METRON-1386 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.5.0 > > > We seem to be missing a fair amount of the required links on our homepage. We > need to add them. > The requirements are listed at > https://www.apache.org/foundation/marks/pmcs#navigation > Reporting is available at: https://whimsy.apache.org/site/project/metron -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1071) Create CONTRIBUTING.md
[ https://issues.apache.org/jira/browse/METRON-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1071: Fix Version/s: 0.5.0 > Create CONTRIBUTING.md > -- > > Key: METRON-1071 > URL: https://issues.apache.org/jira/browse/METRON-1071 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.5.0 > > > The idea is to have a document on contributing to Metron that lives alongside > our code (and we can then move away from the wiki). This document should > have a couple things in it: > * What we look for from code contributions > * How people can actually contribute code > * How people can contribute, even without code (e.g. reviews) > * Helpful things like setting up Travis on personal repos to avoid full > testing time locally. > It should also integrate nicely with the site-book, so just make sure > everything plays nicely. > See: https://github.com/blog/1184-contributing-guidelines -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1485) Upgrade vagrant for dev environments
[ https://issues.apache.org/jira/browse/METRON-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1485: Fix Version/s: 0.5.0 > Upgrade vagrant for dev environments > > > Key: METRON-1485 > URL: https://issues.apache.org/jira/browse/METRON-1485 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > Fix For: 0.5.0 > > > It looks like we are going to be forced into upgrading vagrant based on some > HashiCorp deprecation activities. See: > [https://www.hashicorp.com/blog/terraform-enterprise-saas-has-a-new-address > ] > I'm still digging around for the vagrant commit that fixes this in newer > versions - I just upgraded to the latest and that fixed it. > Currently, if you don't have centos6 or ubuntu14 locally, you see something > like the following: > $ vagrant up > Bringing machine 'node1' up with 'virtualbox' provider... > ==> node1: Box 'centos/6' could not be found. Attempting to find and > install... > node1: Box Provider: virtualbox > node1: Box Version: >= 0 > The box 'centos/6' could not be found or > could not be accessed in the remote catalog. If this is a private > box on HashiCorp's Atlas, please verify you're logged in via > `vagrant login`. Also, please double-check the name. The expanded > URL and error message are shown below: > URL: ["https://atlas.hashicorp.com/centos/6";] > Error: The requested URL returned error: 404 Not Found -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1447) Heap Size Not Set Correctly by MPack for ES 5.x
[
https://issues.apache.org/jira/browse/METRON-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1447:
Fix Version/s: (was: Next + 1)
0.5.0
> Heap Size Not Set Correctly by MPack for ES 5.x
> ---
>
> Key: METRON-1447
> URL: https://issues.apache.org/jira/browse/METRON-1447
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.5.0
>
>
> When setting the heap size in the MPack, the default settings defined by
> Elasticsearch are still carried through. This results in two sets of -Xms and
> -Xmx settings passed to the JVM running Elasticsearch.
> {code}
> [2017-12-04T15:58:12,385][INFO ][o.e.n.Node ] [ubuntu1] JVM arguments
> [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC,
> -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly,
> -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8,
> -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true,
> -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true,
> -Dio.netty.recycler.maxCapacityPerThread=0,
> -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true,
> -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -verbose:gc,
> -Xloggc:/var/log/elasticsearch/elasticsearch_gc.log,
> -XX:-CMSConcurrentMTEnabled, -XX:+PrintGCDateStamps, -XX:+PrintGCDetails,
> -XX:+PrintGCTimeStamps,
> -XX:ErrorFile=/var/log/elasticsearch/elasticsearch_err.log,
> -XX:ParallelGCThreads=8, -Xms512m, -Xmx512m,
> -Des.path.home=/usr/share/elasticsearch]
> {code}
> I am not sure which setting the JVM chooses to keep or if the behavior is
> well-defined in this scenario. Either way, its confusing for a user.
> It seems that in ES 5.x the preferred method of setting the heap size is in
> `/etc/elasticsearch/jvm.options`. This is where the additional settings are
> originating from. If you comment out or delete the settings in this file,
> things work as expected.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1388) update public web site to point at 0.4.2 new release
[ https://issues.apache.org/jira/browse/METRON-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1388: Fix Version/s: (was: 0.4.3) 0.5.0 > update public web site to point at 0.4.2 new release > > > Key: METRON-1388 > URL: https://issues.apache.org/jira/browse/METRON-1388 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley >Assignee: Matt Foley >Priority: Major > Fix For: 0.5.0 > > > Of course also update current-book from 0.4.1 to 0.4.2 documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1350) Add reservoir sampling functions to Stellar
[ https://issues.apache.org/jira/browse/METRON-1350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1350: Fix Version/s: 0.5.0 > Add reservoir sampling functions to Stellar > --- > > Key: METRON-1350 > URL: https://issues.apache.org/jira/browse/METRON-1350 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Sampling capabilities would fit very well with the profiler and enable > algorithms that do not necessarily support our existing probabilistic > sketches. We should add a reservoir sampler and utilities to merge and > resample. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1273) Website documentation link should point to the current site-book
[ https://issues.apache.org/jira/browse/METRON-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1273: Fix Version/s: 0.5.0 > Website documentation link should point to the current site-book > > > Key: METRON-1273 > URL: https://issues.apache.org/jira/browse/METRON-1273 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Trivial > Fix For: 0.5.0 > > > Currently, the metron.apache.org site[1] is pointing to the wiki > documentation page[2], which is no longer the correct place to point for > Metron documentation. We should repoint it to the current site-book. > 1: http://metron.apache.org/documentation/#docshome > 2: https://cwiki.apache.org/confluence/display/METRON/Documentation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1394) Create Rest endpoint to add the ACL for current user to kafka topics
[ https://issues.apache.org/jira/browse/METRON-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1394: Fix Version/s: 0.5.0 > Create Rest endpoint to add the ACL for current user to kafka topics > - > > Key: METRON-1394 > URL: https://issues.apache.org/jira/browse/METRON-1394 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Major > Fix For: 0.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1455) Patch and Replace methods in the REST UpdateController return 400
[
https://issues.apache.org/jira/browse/METRON-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1455:
Fix Version/s: 0.5.0
> Patch and Replace methods in the REST UpdateController return 400
> --
>
> Key: METRON-1455
> URL: https://issues.apache.org/jira/browse/METRON-1455
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Casey Stella
>Priority: Major
> Fix For: 0.5.0
>
>
> A regression was recently introduced where a patch or replace request to the
> REST UpdateController returns a 400 status code. For example, this patch
> request:
> {code:java}
> {
> "guid" : "bro_2",
> "sensorType" : "bro",
> "patch" : [
> {
> "op": "add",
> "path": "/project",
> "value": "metron"
> }
> ]
> }{code}
> works in the UpdateControllerIntegrationTest but returns a 400 in full dev.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1377) Stellar function to generate typosquatted domains (similar to dnstwist)
[ https://issues.apache.org/jira/browse/METRON-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1377: Fix Version/s: 0.5.0 > Stellar function to generate typosquatted domains (similar to dnstwist) > --- > > Key: METRON-1377 > URL: https://issues.apache.org/jira/browse/METRON-1377 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > As a component of a strategy to detect Typosquatting (see > https://en.wikipedia.org/wiki/Typosquatting), generating typosquatted domains > is necessary. As such, a stellar function which replicates the functionality > of dnstwist would be of use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1487) Define Performance Benchmarks for Enrichment Topology
[ https://issues.apache.org/jira/browse/METRON-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1487: Fix Version/s: (was: Next + 1) 0.5.0 > Define Performance Benchmarks for Enrichment Topology > - > > Key: METRON-1487 > URL: https://issues.apache.org/jira/browse/METRON-1487 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.2 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > Define a set of performance benchmarks for the Enrichment topology. These > benchmarks should be repeatable to help detect performance regressions that > might occur. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-941) native PaloAlto parser corrupts message when having a comma in the payload
[
https://issues.apache.org/jira/browse/METRON-941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-941:
---
Fix Version/s: 0.5.0
> native PaloAlto parser corrupts message when having a comma in the payload
> --
>
> Key: METRON-941
> URL: https://issues.apache.org/jira/browse/METRON-941
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.0
> Environment: full-dev master
>Reporter: Christian Tramnitz
>Priority: Minor
> Fix For: 0.5.0
>
>
> When a data field contains a comma (i.e. the URL, not too uncommon), the
> split(",") kicks in and the rest of the message if off by few fields due to
> positional definition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1417) Disable pcap-service by default in Monit
[ https://issues.apache.org/jira/browse/METRON-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1417: Fix Version/s: 0.5.0 > Disable pcap-service by default in Monit > > > Key: METRON-1417 > URL: https://issues.apache.org/jira/browse/METRON-1417 > Project: Metron > Issue Type: Task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.5.0 > > > full dev is starting to burst at the seams, so this is one item with less > frequent use as of late that doesn't need to be running by default. We > already don't run the pcap topology by default, so it seems reasonable to do > this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1337) List of facets should not be hardcoded
[ https://issues.apache.org/jira/browse/METRON-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1337: Fix Version/s: 0.5.0 > List of facets should not be hardcoded > -- > > Key: METRON-1337 > URL: https://issues.apache.org/jira/browse/METRON-1337 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.5.0 > > > Currently the facet fields shown in the left panel of the Alerts UI is hard > coded in a javascript file. This should be configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1380) Create a typosquatting use-case
[ https://issues.apache.org/jira/browse/METRON-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1380: Fix Version/s: 0.5.0 > Create a typosquatting use-case > --- > > Key: METRON-1380 > URL: https://issues.apache.org/jira/browse/METRON-1380 > Project: Metron > Issue Type: New Feature >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > We should have a use-case to support detecting typosquatted domains in stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1345) Update EC2 README for custom Ansible tags
[ https://issues.apache.org/jira/browse/METRON-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1345: Fix Version/s: 0.5.0 > Update EC2 README for custom Ansible tags > - > > Key: METRON-1345 > URL: https://issues.apache.org/jira/browse/METRON-1345 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1393) Fix bro Elasticsearch template
[ https://issues.apache.org/jira/browse/METRON-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1393: Fix Version/s: 0.5.0 > Fix bro Elasticsearch template > -- > > Key: METRON-1393 > URL: https://issues.apache.org/jira/browse/METRON-1393 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.5.0 > > > After the ES 5.6.2 upgrade there were found to be some string types in the > bro template that remained from a prior merge - > https://issues.apache.org/jira/browse/METRON-939. This breaks aggregate > searches via the alerts UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1396) Fix .gitignore files to not ignore themselves
[ https://issues.apache.org/jira/browse/METRON-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1396: Fix Version/s: (was: Next + 1) 0.5.0 > Fix .gitignore files to not ignore themselves > - > > Key: METRON-1396 > URL: https://issues.apache.org/jira/browse/METRON-1396 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Trivial > Fix For: 0.5.0 > > > Two `.gitignore` files have a pattern to ignore hidden files: `.*`. The > problem is that `.gitignore` files match this pattern and are therefore > ignored. An exception should be added for the `.gitignore` itself. > The two files are: > ``` > metron-deployment/packaging/docker/deb-docker/.gitignore > metron-deployment/packaging/docker/rpm-docker/.gitignore > ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1411) Fix sed command in Upgrading.md
[ https://issues.apache.org/jira/browse/METRON-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1411: Fix Version/s: (was: Next + 1) 0.5.0 > Fix sed command in Upgrading.md > --- > > Key: METRON-1411 > URL: https://issues.apache.org/jira/browse/METRON-1411 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > Fix For: 0.5.0 > > > There's an extra pair of single quotes that causes problems on some sed > versions. Just need to drop them, there were extraneous anyway. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1451) On Centos full dev, Metron Indexing shows up as stopped
[
https://issues.apache.org/jira/browse/METRON-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1451:
Fix Version/s: (was: Next + 1)
0.5.0
> On Centos full dev, Metron Indexing shows up as stopped
> ---
>
> Key: METRON-1451
> URL: https://issues.apache.org/jira/browse/METRON-1451
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
> Fix For: 0.5.0
>
>
> In Ambari, the Metron Indexing status is shown as 'stopped'. The agent log
> shows the following error:
> {code:java}
> INFO 2018-02-05 22:21:39,990 PythonReflectiveExecutor.py:67 - Reflective
> command failed with exception:
> Traceback (most recent call last):
> File
> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
> line 59, in run_file
> imp.load_source('__main__', script)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/package/scripts/indexing_master.py",
> line 18, in
> import requests
> File "/usr/lib/python2.6/site-packages/requests/__init__.py", line 53, in
>
> from .packages.urllib3.contrib import pyopenssl
> File "/usr/lib/python2.6/site-packages/requests/packages/__init__.py", line
> 61, in load_module
> if name in sys.modules:
> AttributeError: 'NoneType' object has no attribute 'modules'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1428) Travis build failing from metron-config
[
https://issues.apache.org/jira/browse/METRON-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1428:
Fix Version/s: 0.5.0
> Travis build failing from metron-config
> ---
>
> Key: METRON-1428
> URL: https://issues.apache.org/jira/browse/METRON-1428
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
> Fix For: 0.5.0
>
>
> The build is hiding logs in the metron-config failure and then hanging
> indefinitely. Running the build locally, ie
> {code:java}
> metron/metron-interface/metron-config$ mvn test{code}
> resulted in the following error:
> {code:java}
> [INFO] 24 01 2018 09:13:44.242:INFO [PhantomJS 2.1.1 (Mac OS X 0.0.0)]:
> Connected on socket /#2JMBj2Qu9STI0gMF with id 22733174
> PhantomJS 2.1.1 (Mac OS X 0.0.0): Executed 241 of 241 SUCCESS (1 min 38.478
> secs / 1 min 37.477 secs)
> [INFO] Missing error handler on `socket`.
> [INFO] TypeError: sourceMap.originalPositionFor is not a function
> [INFO] at getMapping
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/remap-istanbul/lib/remap.js:76:25)
> [INFO] at
> /Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/remap-istanbul/lib/remap.js:245:20
> [INFO] at Array.forEach (native)
> [INFO] at
> /Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/remap-istanbul/lib/remap.js:243:37
> [INFO] at Array.forEach (native)
> [INFO] at
> /Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/remap-istanbul/lib/remap.js:192:22
> [INFO] at Array.forEach (native)
> [INFO] at remap
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/remap-istanbul/lib/remap.js:191:12)
> [INFO] at onRunComplete
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma-remap-istanbul/index.js:55:21)
> [INFO] at .
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/lib/events.js:13:22)
> [INFO] at emitTwo (events.js:111:20)
> [INFO] at emit (events.js:191:7)
> [INFO] at emitRunCompleteIfAllBrowsersDone
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/lib/server.js:294:12)
> [INFO] at .
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/lib/server.js:325:7)
> [INFO] at emitOne (events.js:96:13)
> [INFO] at emit (events.js:188:7)
> [INFO] at .
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/lib/server.js:308:12)
> [INFO] at emitTwo (events.js:111:20)
> [INFO] at emit (events.js:191:7)
> [INFO] at onComplete
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/lib/browser.js:143:13)
> [INFO] at Socket.
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/lib/events.js:13:22)
> [INFO] at emitTwo (events.js:111:20)
> [INFO] at Socket.emit (events.js:191:7)
> [INFO] at Socket.onevent
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/node_modules/socket.io/lib/socket.js:335:8)
> [INFO] at Socket.onpacket
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/node_modules/socket.io/lib/socket.js:295:12)
> [INFO] at Client.ondecoded
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/node_modules/socket.io/lib/client.js:193:14)
> [INFO] at Decoder.Emitter.emit
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/component-emitter/index.js:134:20)
> [INFO] at Decoder.add
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/node_modules/socket.io-parser/index.js:247:12)
> [INFO] at Client.ondata
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/node_modules/socket.io/lib/client.js:175:18)
> [INFO] at emitOne (events.js:96:13)
> [INFO] at Socket.emit (events.js:188:7)
> [INFO] at Socket.onPacket
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/node_modules/engine.io/lib/socket.js:101:14)
> [INFO] at emitOne (events.js:96:13)
> [INFO] at WebSocket.emit (events.js:188:7)
> [INFO] at WebSocket.Transport.onPacket
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/node_modules/engine.io/lib/transport.js:104:8)
> [INFO] at WebSocket.Transport.onData
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-config/node_modules/karma/node_modules/engine.io/lib/transport.js:115:8)
> [INFO] at WebSocket.onData
> (/Users/mmiklavcic/devprojects/metron/metron-interface/metron-confi
[jira] [Updated] (METRON-1318) Cannot deploy into EC2
[
https://issues.apache.org/jira/browse/METRON-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1318:
Fix Version/s: 0.5.0
> Cannot deploy into EC2
> --
>
> Key: METRON-1318
> URL: https://issues.apache.org/jira/browse/METRON-1318
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
> Environment: AWS
>Reporter: Paweł Białasiewicz
>Priority: Major
> Fix For: 0.5.0
>
>
> I'm trying to deploy Metron on AWS using the cloud deployment method.
> It looks like it is having problems with building the master node
> {code:java}
> [metron-test] sensors,ambari_master,ec2,monit
> {code}
> I tried rebuilding the node from scratch and running the run script multiple
> times, always with the same result.
> During the installation I'm getting the following log:
> {code:java}
> TASK [setup]
> ***
> ok: [localhost]
> ok: [ec2-34-227-190-43.compute-1.amazonaws.com]
> ok: [ec2-34-201-150-57.compute-1.amazonaws.com]
> ok: [ec2-54-162-174-97.compute-1.amazonaws.com]
> ok: [ec2-34-207-187-45.compute-1.amazonaws.com]
> ok: [ec2-52-91-15-158.compute-1.amazonaws.com]
> ok: [ec2-34-229-153-26.compute-1.amazonaws.com]
> ok: [ec2-52-73-77-81.compute-1.amazonaws.com]
> ok: [ec2-54-175-242-208.compute-1.amazonaws.com]
> ok: [ec2-54-85-210-26.compute-1.amazonaws.com]
> ok: [ec2-34-229-86-210.compute-1.amazonaws.com]
> TASK [metron-builder : Build Metron]
> ***
> fatal: [ec2-34-227-190-43.compute-1.amazonaws.com -> localhost]: FAILED! =>
> {"changed": true, "cmd": "cd
> /root/metron/metron-deployment/amazon-ec2/../playbooks/../.. && mvn clean
> package -DskipTests -T 2C -P HDP-2.5.0.0,mpack", "delta": "0:00:03.308032",
> "end": "2017-11-16 14:04:52.835967", "failed": true, "rc": 1, "start":
> "2017-11-16 14:04:49.527935", "stderr": "", "stdout": "Warning: JAVA_HOME
> environment variable is not set.\n[INFO] Scanning for projects...\n[INFO]
> \n[INFO]
> Reactor Build Order:\n[INFO] \n[INFO] Metron\n[INFO] metron-stellar\n[INFO]
> stellar-common\n[INFO] metron-analytics\n[INFO] metron-maas-common\n[INFO]
> metron-platform\n[INFO] metron-zookeeper\n[INFO]
> metron-test-utilities\n[INFO] metron-integration-test\n[INFO]
> metron-maas-service\n[INFO] metron-common\n[INFO] metron-statistics\n[INFO]
> metron-writer\n[INFO] metron-storm-kafka-override\n[INFO]
> metron-storm-kafka\n[INFO] metron-hbase\n[INFO]
> metron-profiler-common\n[INFO] metron-profiler-client\n[INFO]
> metron-profiler\n[INFO] metron-hbase-client\n[INFO] metron-enrichment\n[INFO]
> m
> etron-indexing\n[INFO] metron-solr\n[INFO] metron-pcap\n[INFO]
> metron-parsers\n[INFO] metron-pcap-backend\n[INFO]
> metron-data-management\n[INFO] metron-api\n[INFO] metron-management\n[INFO]
> elasticsearch-shaded\n[INFO] metron-elasticsearch\n[INFO]
> metron-deployment\n[INFO] Metron Ambari Management Pack\n[INFO]
> metron-contrib\n[INFO] metron-docker\n[INFO] metron-interface\n[INFO]
> metron-config\n[INFO] metron-alerts\n[INFO] metron-rest-client\n[INFO]
> metron-rest\n[INFO] site-book\n[INFO] 3rd party Functions (just for
> tests)\n[INFO] \n[INFO] Using the MultiThreadedBuilder implementation with a
> thread count of 4\n[INFO]
> \n[INFO]
> \n[INFO]
> Building Metron 0.4.2\n[INFO]
> \n[INFO]
> \n[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ Metron
> ---\n[INFO] \n[INFO] --- maven-enforcer-plugin:1.4.1:enforce
> (enforce-versions) @ Metron ---\n[INFO] \n[INFO] ---
> jacoco-maven-plugin:0.7.9:prepare-agent (default) @ Metron ---
> \n[INFO] argLine set to
> -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/root/metron/target/jacoco.exec\n[INFO]
> \n[INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ Metron ---\n[INFO]
> Skipping JaCoCo execution due to missing execution data file.\n[INFO]
> \n[INFO]
> \n[INFO]
> Building metron-stellar 0.4.2\n[INFO]
> \n[INFO]
>
> \n[INFO]
> \n[INFO]
> Building metron-analytics 0.4.2\n[INFO]
> \n[INFO]
>
[jira] [Updated] (METRON-1348) Metron Service Checks Use Wrong Hostname
[
https://issues.apache.org/jira/browse/METRON-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1348:
Fix Version/s: (was: Next + 1)
0.5.0
> Metron Service Checks Use Wrong Hostname
>
>
> Key: METRON-1348
> URL: https://issues.apache.org/jira/browse/METRON-1348
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.5.0
>
>
> The Metron service check can often use the incorrect hostname when checking
> the Alerts UI, Management UI, and REST services.
> Ambari can run the service check on any node in the cluster, not just the
> node the service is actually running on. The service check code currently
> uses the hostname on which the service check is running. If the service is
> not actually installed on that host, the service check will incorrectly fail.
> The service check code should be updated to find the hostname where the
> service is installed and use that hostname.
> For example, here is a log of a service check that is looking on the wrong
> host for the Metron REST service.
> {code}
> 2017-12-08 17:11:30,433 - Checking connectivity to REST application
> 2017-12-08 17:11:30,434 - Checking HTTP connectivity;
> host=hcpua-10.openstacklocal, port=8082, user=metron cmd=curl -sS --max-time
> 3 hcpua-10.openstacklocal:8082
> 2017-12-08 17:11:30,434 - Execute['curl -sS --max-time 3
> hcpua-10.openstacklocal:8082'] {'logoutput': False, 'tries': 3, 'user':
> 'metron', 'try_sleep': 5}
> 2017-12-08 17:11:30,471 - Retrying after 5 seconds. Reason: Execution of
> 'curl -sS --max-time 3 hcpua-10.openstacklocal:8082' returned 7. curl: (7)
> Failed to connect to hcpua-10.openstacklocal port 8082: Connection refused
> 2017-12-08 17:11:35,519 - Retrying after 5 seconds. Reason: Execution of
> 'curl -sS --max-time 3 hcpua-10.openstacklocal:8082' returned 7. curl: (7)
> Failed to connect to hcpua-10.openstacklocal port 8082: Connection refused
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0
[
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1518:
Fix Version/s: (was: Next + 1)
0.5.0
> Build Failure When Using Profile HDP-2.5.0.0
>
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.5.0
>
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
> symbol: class StormTimer
> location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
> private StormTimer expiredFlushTimer;
> ^
> symbol: class StormTimer
> location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
> private StormTimer createTimer(String name) {
> ^
> symbol: class StormTimer
> location: class ProfileBuilderBolt
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1398) Exclude the basic-error-controller from being added to the swagger description
[ https://issues.apache.org/jira/browse/METRON-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1398: Fix Version/s: 0.5.0 > Exclude the basic-error-controller from being added to the swagger description > -- > > Key: METRON-1398 > URL: https://issues.apache.org/jira/browse/METRON-1398 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Minor > Fix For: 0.5.0 > > > exclude the basic-error-controller and web-security-config methods from > being added to the swagger description -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1445) Update performance tuning guide with more explicit parameter instructions
[ https://issues.apache.org/jira/browse/METRON-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1445: Fix Version/s: 0.5.0 > Update performance tuning guide with more explicit parameter instructions > - > > Key: METRON-1445 > URL: https://issues.apache.org/jira/browse/METRON-1445 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1449) Set Zookeeper URL for Stellar Running in Zeppelin Notebook
[ https://issues.apache.org/jira/browse/METRON-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1449: Fix Version/s: (was: Next + 1) 0.5.0 > Set Zookeeper URL for Stellar Running in Zeppelin Notebook > -- > > Key: METRON-1449 > URL: https://issues.apache.org/jira/browse/METRON-1449 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > A user needs to be able to configure a Zookeeper URL for the Stellar session > backing their Zeppelin Notebook. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1379) Add an OBJECT_GET stellar function
[ https://issues.apache.org/jira/browse/METRON-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1379: Fix Version/s: 0.5.0 > Add an OBJECT_GET stellar function > -- > > Key: METRON-1379 > URL: https://issues.apache.org/jira/browse/METRON-1379 > Project: Metron > Issue Type: New Feature >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > With the creation of METRON-1378 we have the ability to create serialized > summaries of data. We need to interact with these summaries from within > Stellar. This enables use-cases like creating a bloom filter of malicious > domains and interacting with that bloom filter from within all the places > where stellar is available. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1467) Replace guava caches in places where the keyspace might be large
[ https://issues.apache.org/jira/browse/METRON-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1467: Fix Version/s: 0.5.0 > Replace guava caches in places where the keyspace might be large > > > Key: METRON-1467 > URL: https://issues.apache.org/jira/browse/METRON-1467 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Based on the performance tuning exercise as part of METRON-1460, guava has > difficulties with cache sizes over 10k. We, unfortunately, are quite > demanding of guava in this regard so we should transition a few uses of guava > to Caffeine: > * Stellar processor cache > * The JoinBolt cache > * The Enrichment Bolt Cache > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1500) Enhance 'prepare-commit' to Support Feature Branches
[ https://issues.apache.org/jira/browse/METRON-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1500: Fix Version/s: (was: Next + 1) 0.5.0 > Enhance 'prepare-commit' to Support Feature Branches > > > Key: METRON-1500 > URL: https://issues.apache.org/jira/browse/METRON-1500 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > Fix For: 0.5.0 > > > I'd like to be able to use `prepare-commit` to review and commit PRs that are > submitted against feature branches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1362: Fix Version/s: (was: Next + 1) 0.5.0 > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.1 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1493) Unhelpful Error Message When Assignment Expressions Fail
[
https://issues.apache.org/jira/browse/METRON-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1493:
Fix Version/s: (was: Next + 1)
0.5.0
> Unhelpful Error Message When Assignment Expressions Fail
>
>
> Key: METRON-1493
> URL: https://issues.apache.org/jira/browse/METRON-1493
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.5.0
>
>
> When running a bad assignment expression in the REPL, the error message
> provides no help as to why the expression fails. It only says that the
> expression fails, which is obvious. This needs to report back to the user
> the root cause of the assignment expression failing.
> {code}
> [Stellar]>>> p := 0/0
> [!] Assignment expression failed
> java.lang.IllegalArgumentException: Assignment expression failed
> at
> org.apache.metron.stellar.common.shell.StellarResult.error(StellarResult.java:115)
> at
> org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:82)
> at
> org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)
> at
> org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:357)
> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1430) Isolate jackson from being used as arguments or returns from JSONUtils
[ https://issues.apache.org/jira/browse/METRON-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1430: Fix Version/s: 0.5.0 > Isolate jackson from being used as arguments or returns from JSONUtils > -- > > Key: METRON-1430 > URL: https://issues.apache.org/jira/browse/METRON-1430 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Currently jackson is used as part of our internal API to JSONUtils. The > problem here is when we shade and relocate jackson. Suddenly we can't use > JSONUtils. We should avoid operating on jackson primitives and rather > provide a convenient wrapper around the places where we use jackson as part > of the API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1302) Split up Indexing Topology into batch and random access sections
[ https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1302: Fix Version/s: 0.5.0 > Split up Indexing Topology into batch and random access sections > > > Key: METRON-1302 > URL: https://issues.apache.org/jira/browse/METRON-1302 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Currently we have the indexing topology handle writing to both random access > indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We > should split these up and configure them separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1370) Create Full Dev Equivalent for Ubuntu
[ https://issues.apache.org/jira/browse/METRON-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1370: Fix Version/s: (was: Next + 1) 0.5.0 > Create Full Dev Equivalent for Ubuntu > - > > Key: METRON-1370 > URL: https://issues.apache.org/jira/browse/METRON-1370 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > With METRON-1351, we will have installable packages to make the installation > of Metron on Ubuntu far simpler. We need to create an equivalent version of > "Full Dev" that runs on Ubuntu. This will allow us to easily ensure that the > packages, along with the rest of Metron, works when run on Ubuntu. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1515) Errors loading stellar functions currently bomb the entire topology, they should be recoverable
[ https://issues.apache.org/jira/browse/METRON-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1515: Fix Version/s: 0.5.0 > Errors loading stellar functions currently bomb the entire topology, they > should be recoverable > --- > > Key: METRON-1515 > URL: https://issues.apache.org/jira/browse/METRON-1515 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > If a bad stellar function gets on the classpath, the entire enrichment > topology (or shell) bombs. We should just log an error and skip the function. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1539) Specialized RENAME field transformer
[ https://issues.apache.org/jira/browse/METRON-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1539: Fix Version/s: 0.5.0 > Specialized RENAME field transformer > > > Key: METRON-1539 > URL: https://issues.apache.org/jira/browse/METRON-1539 > Project: Metron > Issue Type: New Feature >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Currently our advice is to use the Stellar field transformation to do simple > field renaming. Given how common this situation is and how the old field > must be nulled out, thus making the stellar transformation awkward, it's > worthwhile having a specialized field transformer for RENAME. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1521) JSONMapParser is no longer serializable
[ https://issues.apache.org/jira/browse/METRON-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1521: Fix Version/s: 0.5.0 > JSONMapParser is no longer serializable > --- > > Key: METRON-1521 > URL: https://issues.apache.org/jira/browse/METRON-1521 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Currently the JSONMapParser does not function because an anonymous inner > class (TypeRef) is not serializable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1382) Run Stellar in a Zeppelin Notebook
[ https://issues.apache.org/jira/browse/METRON-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1382: Fix Version/s: (was: Next + 1) 0.5.0 > Run Stellar in a Zeppelin Notebook > -- > > Key: METRON-1382 > URL: https://issues.apache.org/jira/browse/METRON-1382 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.2 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > Create a Zeppelin interpreter that allows Stellar to run in a Zeppelin > Notebook, the same way that it runs in the command-line driven REPL. > This should include all of the Stellar "enhancements" available in the REPL > that are not part of the core Stellar language including the following. > * Variable assignment > * Magics > * Docstrings > * Comments > * quit -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1326) Metron deploy with Kerberos fails on Ambari 2.5 during ES service stop
[
https://issues.apache.org/jira/browse/METRON-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1326:
Fix Version/s: 0.5.0
> Metron deploy with Kerberos fails on Ambari 2.5 during ES service stop
> --
>
> Key: METRON-1326
> URL: https://issues.apache.org/jira/browse/METRON-1326
> Project: Metron
> Issue Type: Bug
> Environment: 12 node VM cluster running CentOS 7
>Reporter: Anand Subramanian
>Assignee: Michael Miklavcic
>Priority: Major
> Fix For: 0.5.0
>
>
> I am noticing that Metron deploy is failing when enabling Kerberos on a
> 12-node VM cluster managed by Ambari 2.5.2.
> The error is seen during the "Stop Services" step while kerberizing for
> Elasticsearch Master and Elasticsearch Data Node services.
> I confirmed that the same deployment goes through fine for Ambari 2.4.2
> version. I am able to setup the Kerberized cluster fine.
> For Ambari 2.4, for the "Elasticsearch Data Node Stop" step, we stop the
> slave, and do not check on the status of the service after the 'service stop'
> command was issued. But with Ambari 2.5, we attempt to check the status after
> the service stop command was issued.
> *In Ambari 2.4*
> {code}
> stdout:
> Stop the Slave
> 2017-11-07 10:21:27,755 - Execute['service elasticsearch stop'] {}
> Command completed successfully!
> {code}
> *In Ambari 2.5*
> {code}
> Stop the Slave
> 2017-11-07 10:12:48,481 - Execute['service elasticsearch stop'] {}
> 2017-11-07 10:12:48,599 - Waiting for actual component stop
> Status of the Slave
> 2017-11-07 10:12:48,600 - Execute['service elasticsearch status'] {}
> Command failed after 1 tries
> {code}
> Apparently the status command is returning a result with error code 3, which
> the ambari agent is not liking and hence calling the step as a failure.
> I am not sure entirely if this is something to be handled by Metron or by
> Ambari. Please feel free to close this defect in case this is deemed out of
> scope of Metron.
> Here is the full error log from the UI
> {code}
> stderr:
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py",
> line 71, in
> Elasticsearch().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 332, in execute
> self.execute_prefix_function(self.command_name, 'after', env)
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 350, in execute_prefix_function
> method(env)
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 398, in after_stop
> status_method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py",
> line 59, in status
> Execute(status_cmd)
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 166, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
> line 262, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 72, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 102, in checked_call
> tries=tries, try_sleep=try_sleep,
> timeout_kill_strategy=timeout_kill_strategy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 303, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'service
> elasticsearch status' returned 3. ● elasticsearch.service - Elasticsearch
>Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled;
> vendor preset: disabled)
>Active: inactive (dead)
> Docs: http://www.elastic.co
> Nov 07 10:12:47 metron-12 elasticsearch[25937]: [2017-11-07
> 10:12:47,340][INFO ][cluster.service ] [metron-12.openstacklocal]
> removed
> {{metron-9.openstacklocal}{lTJDzEA6Sp6_6ryTY8XSJQ}{172.22.97.19}{172.22.97.19:9300}{master=false},},
> reason:
> zen-disco-node_left({metron-9.openstacklocal}{
[jira] [Updated] (METRON-1549) Add empty object test to WriterBoltIntegrationTest implementation
[ https://issues.apache.org/jira/browse/METRON-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1549: Fix Version/s: 0.5.0 > Add empty object test to WriterBoltIntegrationTest implementation > - > > Key: METRON-1549 > URL: https://issues.apache.org/jira/browse/METRON-1549 > Project: Metron > Issue Type: Test >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.5.0 > > > Verify expected output when empty objects returned by custom parsers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1516) Support for Ansible 2.5.0
[ https://issues.apache.org/jira/browse/METRON-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1516: Fix Version/s: 0.5.0 > Support for Ansible 2.5.0 > - > > Key: METRON-1516 > URL: https://issues.apache.org/jira/browse/METRON-1516 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 0.5.0 > > > I was forced to update ansible to 2.5.0 to resolve errors with vagrant, but > our ansible scripts don't support it. > > I have 'patch' tested and had no issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1494) Profiler Emits Messages to Kafka When Not Needed
[ https://issues.apache.org/jira/browse/METRON-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1494: Fix Version/s: (was: Next + 1) 0.5.0 > Profiler Emits Messages to Kafka When Not Needed > > > Key: METRON-1494 > URL: https://issues.apache.org/jira/browse/METRON-1494 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.2 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > Using the 'result/triage' expression allows you to send profile data to > Kafka. This allows you to leverage the Threat Triage functionality against > data coming out of the Profiler. > If there is no 'result/triage' expression, then nothing should be sent to > Kafka. Currently, a message containing some data, but no actual profile > value, is sent to Kafka. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1378) Create a summarizer
[
https://issues.apache.org/jira/browse/METRON-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1378:
Fix Version/s: 0.5.0
> Create a summarizer
> ---
>
> Key: METRON-1378
> URL: https://issues.apache.org/jira/browse/METRON-1378
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>Priority: Major
> Fix For: 0.5.0
>
>
> We have a nice and generalized infrastructure for loading data into HBase and
> interacting with it via `flatfile_loader.sh` and `ENRICHMENT_GET()`. It is
> also useful to summarize a set of data into a static data structure, store it
> on HDFS and interact with it via stellar. To this end, to complement the
> `flatfile_loader.sh`, we should have a `flatfile_summarizer.sh` that, using
> the same extractor config, will process a flat file and output a serialized
> object.
> The usecase for this is as follows:
> Let's say that I have a static list of domains in the second column of a CSV,
> domains.csv, and I want to generate a bloom filter with those domains in them
> sans TLD.
> I should be able to create a file called `bloom.ser` with the serialized
> bloom filter given the extractor config:
> {code}
> {
> "config" : {
> "columns" : {
>"rank" : 0,
>"domain" : 1
> },
> "value_transform" : {
>"domain" : "DOMAIN_REMOVE_TLD(domain)"
> },
> "value_filter" : "LENGTH(domain) > 0",
> "state_init" : "BLOOM_INIT()",
> "state_update" : {
>"state" : "BLOOM_ADD(state, domain)"
> },
> "state_merge" : "BLOOM_MERGE(states)",
> "separator" : ","
> },
> "extractor" : "CSV"
> }
> {code}
> Note, the associated stellar function `OBJECT_GET` is pending.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1461) MIN MAX stellar function should take a stats or list object and return min/max
[ https://issues.apache.org/jira/browse/METRON-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1461: Fix Version/s: 0.5.0 > MIN MAX stellar function should take a stats or list object and return min/max > -- > > Key: METRON-1461 > URL: https://issues.apache.org/jira/browse/METRON-1461 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Minor > Fix For: 0.5.0 > > > Currently the MIN and MAX stellar function accepts only the list as input > and the list may only contain objects that are mutually comparable / ordinal. > Modify the method to take a stats or list object and return min/max. > * [|https://hortonworks.jira.com/secure/AddComment!default.jspa?id=163485] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1392) Fix a test case to expect an Exception when replication factor more than number of brokers while creating topic
[ https://issues.apache.org/jira/browse/METRON-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1392: Fix Version/s: 0.5.0 > Fix a test case to expect an Exception when replication factor more than > number of brokers while creating topic > --- > > Key: METRON-1392 > URL: https://issues.apache.org/jira/browse/METRON-1392 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Minor > Fix For: 0.5.0 > > > KafkaServiceImplTest.createTopicShouldFailIfReplicationFactorIsGreaterThanAvailableBrokers() > is suppose to test create topic with replication factor more than number of > brokers, which is expected to throw exception, but the test case doesn't > expect exception and neither mocked to throw exception when the replication > factor is greater than number of brokers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1442) Rest endpoints for operations on Indexing topology is broken after splitting the indexing topology
[ https://issues.apache.org/jira/browse/METRON-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1442: Fix Version/s: 0.5.0 > Rest endpoints for operations on Indexing topology is broken after splitting > the indexing topology > -- > > Key: METRON-1442 > URL: https://issues.apache.org/jira/browse/METRON-1442 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Major > Fix For: 0.5.0 > > > Rest endpoints for operations on Indexing topology is broken after splitting > the indexing topology into random_access_indexing and batch_indexing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1488) user_settings hbase table does not have acls set up for kerberos
[ https://issues.apache.org/jira/browse/METRON-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1488: Fix Version/s: 0.5.0 > user_settings hbase table does not have acls set up for kerberos > > > Key: METRON-1488 > URL: https://issues.apache.org/jira/browse/METRON-1488 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Currently some REST calls will fail because we do not set ACL's on the new > user_settings table, which is new. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1431) Add REGEXP_REPLACE function to Stellar
[ https://issues.apache.org/jira/browse/METRON-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1431: Fix Version/s: 0.5.0 > Add REGEXP_REPLACE function to Stellar > -- > > Key: METRON-1431 > URL: https://issues.apache.org/jira/browse/METRON-1431 > Project: Metron > Issue Type: Improvement >Reporter: 1havran >Priority: Minor > Fix For: 0.5.0 > > > Add REGEXP_REPLACE(input, pattern, value) function to Stellar that will > replace all occurrences of regex pattern within the input by provided value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-777) Create a plugin system for Metron based on 'NAR'
[ https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-777: --- Fix Version/s: 0.5.0 > Create a plugin system for Metron based on 'NAR' > > > Key: METRON-777 > URL: https://issues.apache.org/jira/browse/METRON-777 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 0.5.0 > > > The success of the Metron project will be greatly dependent on community > participation, and with that the ability to adapt and extend Metron without > having to maintain a fork of the project. > As organizations and individuals look to extend the Metron system with custom > parsers, enrichments, and stellar functions that may be proprietary in > nature, the ability to develop and deploy these extensions outside the Metron > code base is critically important. > To that end, and after community discussion and proposal we create or > formalize the 'plugin' development story in Metron. > The proposal is to adapt the Apache Nifi NAR system for use in Metron. This > will provide the system with: > * archetype(s) for developer projects and independent development > * defined packaging and metadata for 'plugin' products > * loading and instantiation with classloader isolation capabilities > * removing the necessity for shading plugin jars > These capabilities will also enable other features, such as plugin lifecycle, > plugin configuration+redeployment, and other things. > The plugin archetypes and their installation will be a followon -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-590) Enable Use of Event Time in Profiler
[ https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-590: --- Fix Version/s: (was: Next + 1) 0.5.0 > Enable Use of Event Time in Profiler > > > Key: METRON-590 > URL: https://issues.apache.org/jira/browse/METRON-590 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.2 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > There are at least two different times that are important to consider when > handling the telemetry messages received by Metron. > (1) Processing time is the time at which Metron processed the message. > (2) Event time is the time at which the event actually occurred. > If Metron is consuming live data and all is well, the processing and event > times may remain close and consistent. When processing time differs from > event time the data produced by the Profiler may be inaccurate. There are a > few scenarios under which these times might differ greatly which would > negatively impact the feature set produced by the Profiler. > (1) When the system has experienced an outage, for example, a scheduled > maintenance window. When restarted a high volume of messages will need to be > processed by the Profiler. The output of the Profiler will indicate an > increase in activity, although no change in activity actually occurred on the > target network. This could happen whether the outage was Metron itself or an > upstream system that feeds data to Metron. > (2) If the user attempts to replay historical telemetry through the Profiler, > the Profiler will attribute the activity to the time period in which it was > processed. Obviously the activity should be attributed to the time period in > which the raw telemetry events originated in. > There are some scenarios when processing time might be preferred and other > use cases where event time is preferred. The Profiler should be enhanced to > allow it to produce profiles based on either processing time or event time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1551) Profiler Should Not Use Java Serialization
[
https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1551:
Fix Version/s: 0.5.0
> Profiler Should Not Use Java Serialization
> --
>
> Key: METRON-1551
> URL: https://issues.apache.org/jira/browse/METRON-1551
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.5.0
>
>
> When running the Profiler in a topology where serialization occurs, the
> following error happens. This can occur when the number of workers is
> greater than 1.
> The topology should not be using Java serialization for serializing tuple
> values as this will negatively impact performance.
> {code}
> 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR]
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.NotSerializableException:
> org.apache.metron.common.configuration.profiler.ProfileResult
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
> org.apache.metron.common.configuration.profiler.ProfileResult
> at
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> ... 6 more
> Caused by: java.io.NotSerializableException:
> org.apache.metron.common.configuration.profiler.ProfileResult
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> ~[?:1.8.0_162]
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> ~[?:1.8.0_162]
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> ~[?:1.8.0_162]
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> ~[?:1.8.0_162]
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> ~[?:1.8.0_162]
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> ~[?:1.8.0_162]
> at
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesSerializer.serializ
[jira] [Updated] (METRON-1384) Increment master version number to 0.4.3 for on-going development
[ https://issues.apache.org/jira/browse/METRON-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1384: Fix Version/s: (was: 0.4.2) 0.5.0 > Increment master version number to 0.4.3 for on-going development > - > > Key: METRON-1384 > URL: https://issues.apache.org/jira/browse/METRON-1384 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley >Assignee: Matt Foley >Priority: Major > Fix For: 0.5.0 > > > ...now that 0.4.2 is being released to public. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1376) RC Check Script should have named parameters
[ https://issues.apache.org/jira/browse/METRON-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1376: Fix Version/s: 0.5.0 > RC Check Script should have named parameters > > > Key: METRON-1376 > URL: https://issues.apache.org/jira/browse/METRON-1376 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 0.5.0 > > > The script should actually have opt type parameter support -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1374) Script the RC checking process
[ https://issues.apache.org/jira/browse/METRON-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1374: Fix Version/s: 0.5.0 > Script the RC checking process > -- > > Key: METRON-1374 > URL: https://issues.apache.org/jira/browse/METRON-1374 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 0.5.0 > > > There are several steps to validating a release for metron. > These can be scripted, such that they are easier to run. > This script should: > * create a working directory > * download the KEYS > * download the release files ( metron and bro plugin ) as asc files > * add the keys > * validate the downloaded files > * optionally build and test metron, including rpm builds > * optionally start full_dev -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1543) Unable to Set Parser Output Topic in Sensor Config
[ https://issues.apache.org/jira/browse/METRON-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1543: Fix Version/s: (was: Next + 1) 0.5.0 > Unable to Set Parser Output Topic in Sensor Config > -- > > Key: METRON-1543 > URL: https://issues.apache.org/jira/browse/METRON-1543 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > The only way to alter the output topic for a Parser topology is to manually > launch the topology using the CLI with the `-ot` parameter. > The user needs to be able to define this as part of the sensor's parser > configuration so that the value is stored in Zookeeper and can be altered in > the Management UI and launched from Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1439) Turn off git pager in platform-info script
[ https://issues.apache.org/jira/browse/METRON-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1439: Fix Version/s: (was: Next + 1) 0.5.0 > Turn off git pager in platform-info script > -- > > Key: METRON-1439 > URL: https://issues.apache.org/jira/browse/METRON-1439 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Trivial > Fix For: 0.5.0 > > > Add the --no-pager arg to the relevant git commands to both print it to the > output and avoid the user having to manually exit the git command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1452) Rebase Dev Environment on Latest CentOS 6
[ https://issues.apache.org/jira/browse/METRON-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1452: Fix Version/s: (was: Next + 1) 0.5.0 > Rebase Dev Environment on Latest CentOS 6 > - > > Key: METRON-1452 > URL: https://issues.apache.org/jira/browse/METRON-1452 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.2 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > Currently the CentOS development environment > (`metron-deployment/development/centos6`) is based on an image > [metron/centos_base|https://app.vagrantup.com/metron/boxes/centos_base] that > has not been updated in 11 months. This image is really just a snapshot of > [bento/centos6.7|https://app.vagrantup.com/bento/boxes/centos-6.7] from 11 > months ago. The bento/centos6.7 image has not been updated in quite some time > also. > On the other hand, the [centos/6|https://app.vagrantup.com/centos/boxes/6] > image was updated 23 days ago. Presumably these images are receiving critical > patches for long term support. > We should base the CentOS development environment > `metron-deployment/development/centos6` on the > [centos/6|https://app.vagrantup.com/centos/boxes/6] image so that we can be > confident that Metron continues to work on the latest patches for the CentOS > 6 series. > This would match what we do for the Ubuntu development environment which is > based on [ubuntu/trusty64|https://app.vagrantup.com/ubuntu/boxes/trusty64]. > This image continues to receive updates regularly despite the age of the > Ubuntu 14 release. It was updated just 3 days ago. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1529) CONFIG_GET Fails to Retrieve Latest Config When Run in Zeppelin REPL
[
https://issues.apache.org/jira/browse/METRON-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1529:
Fix Version/s: (was: Next + 1)
0.5.0
> CONFIG_GET Fails to Retrieve Latest Config When Run in Zeppelin REPL
>
>
> Key: METRON-1529
> URL: https://issues.apache.org/jira/browse/METRON-1529
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
> Fix For: 0.5.0
>
>
> The configuration values retrieve by CONFIG_GET are incorrect when run in the
> Zeppelin REPL. The cache backing CONFIG_GET retrieves the correct value when
> the function is initialized. If the value is changed either in the same
> session or by an external process, the cache is never updated to the latest
> value. Restarting the Zeppelin REPL session and forcing reinitialization
> will cause the correct, latest value to be retrieved.
> Do the following in a Zeppelin Notebook to replicate the bug.
> 1. Follow the README to install the Stellar Zeppelin Interpreter.
> 2. In Zeppelin > Interpreters > Stellar, set the zookeeper URL property.
> {code}
> zookeeper.url = localhost:2181
> {code}
> 3. In Zeppelin > Interpreters > Stellar, set the following additional
> dependencies.
> | artifact | exclude
> |
> | org.apache.metron:metron-management:0.4.3 |
> |
> | org.apache.metron:metron-common:0.4.3 |
> |
> | io.thekraken:grok:0.1.0 |
> org.apache.commons:commons-lang3 |
> | org.apache.commons:commons-lang3:3.2 |
> |
>
>
> 4. Create a notebook and run the following.
> {code}
> CONFIG_GET("GLOBAL")
> {
> "k6" : "v6"
> }
> {code}
> {code}
> CONFIG_PUT("GLOBAL", '{ "k7":"v7" }')
> {code}
> {code}
> CONFIG_GET("GLOBAL")
> {
> "k6" : "v6"
> }
> {code}
> 5. The last result shold be "k7"/"v7", but is instead the old value "k6"/"v6".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1530) Default proxy config settings in metron-contrib need to be updated
[ https://issues.apache.org/jira/browse/METRON-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1530: Fix Version/s: 0.5.0 > Default proxy config settings in metron-contrib need to be updated > -- > > Key: METRON-1530 > URL: https://issues.apache.org/jira/browse/METRON-1530 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Priority: Minor > Fix For: 0.5.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The current default target settings inside of metron-config's proxy.conf.json > file result in Gateway Timeout errors in the browser. I was able to resolve > this by matching the settings inside of metron-alerts' proxy file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1511) Unable to Serialize Profiler Configuration
[ https://issues.apache.org/jira/browse/METRON-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1511: Fix Version/s: (was: 0.4.3) 0.5.0 > Unable to Serialize Profiler Configuration > -- > > Key: METRON-1511 > URL: https://issues.apache.org/jira/browse/METRON-1511 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > The Profiler configuration objects `ProfileConfig` and `ProfilerConfig` can > be deserialized, but they cannot be serialized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1443) Missing Critical MPack Install Instruction for Ubuntu
[ https://issues.apache.org/jira/browse/METRON-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1443: Fix Version/s: (was: Next + 1) 0.5.0 > Missing Critical MPack Install Instruction for Ubuntu > - > > Key: METRON-1443 > URL: https://issues.apache.org/jira/browse/METRON-1443 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.5.0 > > > When installing Elasticsearch with the MPack on Ubuntu, you must manually > install the Elasticsearch repositories. The Mpack itself does not do this, > like it does on CentOS. > When the development environment on Ubuntu is spun-up this step is performed > within Ansible as a prerequisite to the Mpack install. Until this can be > fixed so that it matches what happens in CentOS this needs to be at least > documented. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1483) Create a tool to monitor performance of the topologies
[ https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1483: Fix Version/s: 0.5.0 > Create a tool to monitor performance of the topologies > -- > > Key: METRON-1483 > URL: https://issues.apache.org/jira/browse/METRON-1483 > Project: Metron > Issue Type: New Feature >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > In performance evaluation, generating synthetic load and monitoring the write > throughput of our kafka-to-kafka topologies has required a lot of custom > scripting. We should have a tool that could do the following: > * Take a file representing a message template and generate synthetic load at > a given events per second > * Monitor the kafka offsets of a topic and report throughput numbers in > events per second > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1471) Migrate shuffle connections to local or shuffle
[ https://issues.apache.org/jira/browse/METRON-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1471: Fix Version/s: 0.5.0 > Migrate shuffle connections to local or shuffle > --- > > Key: METRON-1471 > URL: https://issues.apache.org/jira/browse/METRON-1471 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.5.0 > > > Currently, we use shuffle groupings when we do not want to group by field. > We should, instead, use local or shuffle groupings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
