[GitHub] metron issue #1015: METRON-1544 Flaky test: org.apache.metron.stellar.common...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1015 Is there not some relevant Caffeine test we can ape for this? ---
[jira] [Commented] (METRON-1544) Flaky test: org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
[
https://issues.apache.org/jira/browse/METRON-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477236#comment-16477236
]
ASF GitHub Bot commented on METRON-1544:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1015
Is there not some relevant Caffeine test we can ape for this?
> Flaky test:
> org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
>
>
> Key: METRON-1544
> URL: https://issues.apache.org/jira/browse/METRON-1544
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
> Environment: #uname -a
> Linux 60a83dc7a2ce 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Pravin Dsilva
>Assignee: Nick Allen
>Priority: Major
>
> command used: mvn test
> The test fails intermittently on master branch.
> {code:java}
> Running org.apache.metron.stellar.common.CachingStellarProcessorTest
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec <<<
> FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest
> testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest)
> Time elapsed: 0.053 sec <<< FAILURE!
> java.lang.AssertionError: expected:<6> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.stellar.common.CachingStellarProcessorTest.testCaching(CachingStellarProcessorTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1015: METRON-1544 Flaky test: org.apache.metron.stellar.common...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1015 > @ben-manes Youâre code shouldnât make assumptions on what is evicted as we will try to maximize the hit rate using various techniques. That makes a ton of sense. Thanks for the tips. ---
[jira] [Commented] (METRON-1544) Flaky test: org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
[
https://issues.apache.org/jira/browse/METRON-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477268#comment-16477268
]
ASF GitHub Bot commented on METRON-1544:
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1015
> @ben-manes You’re code shouldn’t make assumptions on what is evicted as
we will try to maximize the hit rate using various techniques.
That makes a ton of sense. Thanks for the tips.
> Flaky test:
> org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
>
>
> Key: METRON-1544
> URL: https://issues.apache.org/jira/browse/METRON-1544
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
> Environment: #uname -a
> Linux 60a83dc7a2ce 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Pravin Dsilva
>Assignee: Nick Allen
>Priority: Major
>
> command used: mvn test
> The test fails intermittently on master branch.
> {code:java}
> Running org.apache.metron.stellar.common.CachingStellarProcessorTest
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec <<<
> FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest
> testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest)
> Time elapsed: 0.053 sec <<< FAILURE!
> java.lang.AssertionError: expected:<6> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.stellar.common.CachingStellarProcessorTest.testCaching(CachingStellarProcessorTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1015: METRON-1544 Flaky test: org.apache.metron.stellar.common...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1015 In the latest, I removed the problematic test case. There is no need to test the eviction policy of the underlying cache, which is effectively what is happening. If anyone can find gaps in test coverage, please let me know. ---
[jira] [Commented] (METRON-1544) Flaky test: org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
[
https://issues.apache.org/jira/browse/METRON-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477301#comment-16477301
]
ASF GitHub Bot commented on METRON-1544:
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1015
In the latest, I removed the problematic test case. There is no need to
test the eviction policy of the underlying cache, which is effectively what is
happening. If anyone can find gaps in test coverage, please let me know.
> Flaky test:
> org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
>
>
> Key: METRON-1544
> URL: https://issues.apache.org/jira/browse/METRON-1544
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
> Environment: #uname -a
> Linux 60a83dc7a2ce 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Pravin Dsilva
>Assignee: Nick Allen
>Priority: Major
>
> command used: mvn test
> The test fails intermittently on master branch.
> {code:java}
> Running org.apache.metron.stellar.common.CachingStellarProcessorTest
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec <<<
> FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest
> testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest)
> Time elapsed: 0.053 sec <<< FAILURE!
> java.lang.AssertionError: expected:<6> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.stellar.common.CachingStellarProcessorTest.testCaching(CachingStellarProcessorTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (METRON-1566) Alert updates are not propagated to metaalert child alerts
Ryan Merriman created METRON-1566: - Summary: Alert updates are not propagated to metaalert child alerts Key: METRON-1566 URL: https://issues.apache.org/jira/browse/METRON-1566 Project: Metron Issue Type: Bug Reporter: Ryan Merriman This can be reproduced as follows: * Create a metaalert from an alert * Update that alert with the patch endpoint * The change to the alert is not reflected in the metaalert child alert -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1016: METRON-1564: Full dev kafka has offsets.topic.rep...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1016 ---
[jira] [Commented] (METRON-1564) Full dev kafka has offsets.topic.replication.factor set to 3 instead of 1
[ https://issues.apache.org/jira/browse/METRON-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477529#comment-16477529 ] ASF GitHub Bot commented on METRON-1564: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1016 > Full dev kafka has offsets.topic.replication.factor set to 3 instead of 1 > - > > Key: METRON-1564 > URL: https://issues.apache.org/jira/browse/METRON-1564 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > I saw an issue where no data was flowing through any topologies. A repeated > error in kafka showed offsets.topic.replication.factor being too low. We > should default this on full dev to 1 to avoid this issue. > On further investigation, it looks like this setting changes between kafka > versions and will no longer auto create the offset topic on some versions if > the replication factor is too high, preventing data from flowing through. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1018: METRON-1566: Alert updates are not propagated to ...
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/1018
METRON-1566: Alert updates are not propagated to metaalert child alerts
## Contributor Comments
Updates to alerts will now be reflected in metaalert child alerts. This
was caused by the ElasticsearchMetaAlertDao class calling the update method on
IndexDao directly rather than calling it's own update method. Steps to
reproduce:
1. Find an alert.
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"fields": [
"guid"
],
"from": 0,
"indices": [
"snort"
],
"query": "*",
"size": 1
}' 'http://node1:8082/api/v1/search/search'
```
2. Create a metaalert from the alert in step 1.
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"alerts": [
{
"guid": "dc4abb51-3b7e-4875-b3af-a03b5fd783d9",
"index": "snort",
"sensorType": "snort"
}
],
"groups": [
"test"
]
}' 'http://node1:8082/api/v1/metaalert/create'
```
3. Update the original alert.
```
curl -u user:password -X PATCH --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "dc4abb51-3b7e-4875-b3af-a03b5fd783d9",
"index": "snort",
"patch": [
{
"op": "add"
, "path": "/newField"
, "value": "value"
}
],
"sensorType": "snort"
}' 'http://node1:8082/api/v1/update/patch'
```
4. Verify the change was applied to the alert. The "newField" field should
be present.
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "dc4abb51-3b7e-4875-b3af-a03b5fd783d9",
"index": "snort",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```
5. Verify the change was also applied to the child alert in the metaalert.
The "newField" field should also be present in the child alert.
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "a76e0bae-4420-4f31-b2b3-234cd9fb9e60",
"index": "metaalert",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
I hit another bug when testing that resulted from timestamp being set to 0
when updating metaalert objects. Since it was keeping this use case from
working I added the fix to this PR. I also added a change to HBaseDao to guard
against this in the future and a test to go with it.
I verified this in full dev.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you e
[jira] [Commented] (METRON-1566) Alert updates are not propagated to metaalert child alerts
[
https://issues.apache.org/jira/browse/METRON-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478040#comment-16478040
]
ASF GitHub Bot commented on METRON-1566:
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/1018
METRON-1566: Alert updates are not propagated to metaalert child alerts
## Contributor Comments
Updates to alerts will now be reflected in metaalert child alerts. This
was caused by the ElasticsearchMetaAlertDao class calling the update method on
IndexDao directly rather than calling it's own update method. Steps to
reproduce:
1. Find an alert.
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"fields": [
"guid"
],
"from": 0,
"indices": [
"snort"
],
"query": "*",
"size": 1
}' 'http://node1:8082/api/v1/search/search'
```
2. Create a metaalert from the alert in step 1.
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"alerts": [
{
"guid": "dc4abb51-3b7e-4875-b3af-a03b5fd783d9",
"index": "snort",
"sensorType": "snort"
}
],
"groups": [
"test"
]
}' 'http://node1:8082/api/v1/metaalert/create'
```
3. Update the original alert.
```
curl -u user:password -X PATCH --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "dc4abb51-3b7e-4875-b3af-a03b5fd783d9",
"index": "snort",
"patch": [
{
"op": "add"
, "path": "/newField"
, "value": "value"
}
],
"sensorType": "snort"
}' 'http://node1:8082/api/v1/update/patch'
```
4. Verify the change was applied to the alert. The "newField" field should
be present.
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "dc4abb51-3b7e-4875-b3af-a03b5fd783d9",
"index": "snort",
"sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```
5. Verify the change was also applied to the child alert in the metaalert.
The "newField" field should also be present in the child alert.
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d '{
"guid": "a76e0bae-4420-4f31-b2b3-234cd9fb9e60",
"index": "metaalert",
"sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```
I hit another bug when testing that resulted from timestamp being set to 0
when updating metaalert objects. Since it was keeping this use case from
working I added the fix to this PR. I also added a change to HBaseDao to guard
against this in the future and a test to go with it.
I verified this in full dev.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apac
[GitHub] metron pull request #1019: METRON-1555: Update REST to run YARN and MR jobs
GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1019 METRON-1555: Update REST to run YARN and MR jobs ## Contributor Comments This PR sets us up to run YARN and MR jobs inside our REST application. Changes include: - addition of maven dependencies - addition of -Dhdp.version parameter to the REST startup script - MPack now supplies the hdp.version parameter - MPack now sets up a "metron" service user HDFS directory needed for running MR jobs - MPack now sets up a pcap HDFS directory - addition of a sample Pcap controller and service to demonstrate running MR jobs in REST This should work in full dev as is. All you need to do is generate some pcap data and hit the pcap endpoint included in this PR. It should return raw binary pcap data. It is assumed the PcapContoller, PcapService, PcapServiceImpl and PcapResponse classes will all be replaced or significantly changed in future tasks. There are presented here temporarily as examples. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron pcap-rest-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1019.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1019 commit 4dcee51dd544e6c064dcc9dd1478c923a00c8281 Author: merrimanr Date: 2018-05-07T20:52:09Z added simple pcap endpoint to rest commit 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1 Author: merrimanr Date: 2018-05-08T22:32:03Z pcap query runs in rest ---
[jira] [Commented] (METRON-1555) Update REST to run YARN and MR jobs
[ https://issues.apache.org/jira/browse/METRON-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478072#comment-16478072 ] ASF GitHub Bot commented on METRON-1555: GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/1019 METRON-1555: Update REST to run YARN and MR jobs ## Contributor Comments This PR sets us up to run YARN and MR jobs inside our REST application. Changes include: - addition of maven dependencies - addition of -Dhdp.version parameter to the REST startup script - MPack now supplies the hdp.version parameter - MPack now sets up a "metron" service user HDFS directory needed for running MR jobs - MPack now sets up a pcap HDFS directory - addition of a sample Pcap controller and service to demonstrate running MR jobs in REST This should work in full dev as is. All you need to do is generate some pcap data and hit the pcap endpoint included in this PR. It should return raw binary pcap data. It is assumed the PcapContoller, PcapService, PcapServiceImpl and PcapResponse classes will all be replaced or significantly changed in future tasks. There are presented here temporarily as examples. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron pcap-rest-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1019.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1019 commit 4dcee51dd544e6c064dcc9dd1478c923a00c8281 Author: merrimanr Date: 2018-05-07T20:52:09Z added simple pcap endpoint to rest commit 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1 Author: merrimanr Date: 2018-05-08T22:32:03Z pcap query runs in rest > Update REST to run YARN and MR jobs > --- > > Key: METRON-1555 > URL: https://issues.apache.org/jira/browse/METRON-1555 > Project: Metron > Issue
[jira] [Commented] (METRON-1555) Update REST to run YARN and MR jobs
[ https://issues.apache.org/jira/browse/METRON-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478415#comment-16478415 ] ASF GitHub Bot commented on METRON-1555: Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1019 Based on the temporary/example state of some of this PR would it make sense to explicitly keep this one out of the upcoming release, regardless of the review cycle? > Update REST to run YARN and MR jobs > --- > > Key: METRON-1555 > URL: https://issues.apache.org/jira/browse/METRON-1555 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > This task involves enabling REST to submit YARN or MR jobs. We will likely > need to: > * update Maven dependencies to include YARN and MR libraries in the > classpath and resolve any version conflicts > * update REST start script to include properties required for YARN > * update the MPack for any additional setup work (create user HDFS directory > for example) and properties needed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1019: METRON-1555: Update REST to run YARN and MR jobs
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1019 Based on the temporary/example state of some of this PR would it make sense to explicitly keep this one out of the upcoming release, regardless of the review cycle? ---
