[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 Doesn't necessarily have to be Docker but yes you would have to start something up. That step is now outside of the integration tests. We could keep the current in memory code and offer

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 So, debugging the integration tests will require having docker running in the background now? ---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/851 ---

[GitHub] metron issue #859: METRON-1345: Update EC2 README for custom Ansible tags

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/859 +1 Thanks! ---

[GitHub] metron issue #851: METRON-1336 Patching Can Result in Bad Configuration

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/851 Thanks guys. Nice work on the meta-PRs. I like how that turned out. ---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 @ottobackwards I also added a way to run the e2e tests locally. Once your Docker app is up and running locally, get the $DOCKER_HOST ip address and replace "localhost" with that ip address in

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 @justinleet latest commit includes the changes I had to make to switch a preexisting integration test to the new infrastructure. I verified it works locally, we'll see how it goes with Travis.

[GitHub] metron issue #851: METRON-1336 Patching Can Result in Bad Configuration

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/851 +1 by inspection. Nice work Nick. ---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/851#discussion_r155347462 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java --- @@ -572,15 +590,22 @@ public

[GitHub] metron issue #851: METRON-1336 Patching Can Result in Bad Configuration

Github user cestella commented on the issue: https://github.com/apache/metron/pull/851 I'm comfortable with this. +1 by inspection ---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 Sure a discussion on that topic is fine with me. I'm referring to the existing integration tests. ---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 Any chance we can have a discuss on moving other integration tests over? You mean the regular tests right, like profiler etc? ---

[GitHub] metron pull request #859: METRON-1345: Update EC2 README for custom Ansible ...

GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/859 METRON-1345: Update EC2 README for custom Ansible tags https://issues.apache.org/jira/browse/METRON-1345 Simple improvement/correction to the EC2 README. This runs the command with the

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 @justinleet I think that is a great idea. I didn't do it initially to keep the size of this PR modest but I can add it in (we can always roll it back and move it to another PR if we want). I'll

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 Sorry @ottobackwards, you're right the instructions are a little vague. First, `metron-contrib/metron-docker-e2e/compose` is the directory you should be in. I should have added that at the

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/858#discussion_r155339646 --- Diff: metron-interface/metron-alerts/pom.xml --- @@ -130,4 +146,44 @@ + + + + e2e

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/858#discussion_r155339627 --- Diff: metron-interface/metron-alerts/e2e/utils/e2e_util.ts --- @@ -46,25 +48,53 @@ export function waitForStalenessOf (_element ) { }

[GitHub] metron issue #851: METRON-1336 Patching Can Result in Bad Configuration

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/851 Bump. Anything else we need on this one? ---

[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

Github user iraghumitra commented on the issue: https://github.com/apache/metron/pull/857 @merrimanr renamed the variable POLLING_DEFAULT_STATE to DISABLE_POLLING ---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 @merrimanr can you add what directory you should be in when you run the commands? ---

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/858#discussion_r155288566 --- Diff: metron-interface/metron-alerts/pom.xml --- @@ -130,4 +146,44 @@ + + + +

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/858#discussion_r155288298 --- Diff: metron-interface/metron-alerts/e2e/utils/e2e_util.ts --- @@ -46,25 +48,53 @@ export function waitForStalenessOf (_element ) { }

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/858 @merrimanr Did you look into what migrating our non-e2e tests integration tests would involve? I think for a POC, it's important to get a sense of how we'd be able to unify the infrastructure of

[GitHub] metron issue #854: Experimental Improvements - Feedback Only - Do Not Merge

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/854 Casey and I investigated this previously during one of the times we were running into the upper time limit. The main reason we ultimately chose not to do this was because we share the

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 Wow, this is awesome. I'm looking forward to getting into it. First thing that pops to mind is how this relates to or is effected by @nickwallen's https://github.com/apache/metron/pull/854

[GitHub] metron pull request #857: METRON-1340: Improve e2e tests for metron alerts

Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/857#discussion_r155261159 --- Diff: metron-interface/metron-alerts/src/app/utils/constants.ts --- @@ -38,5 +38,6 @@ export let TREE_SUB_GROUP_SIZE = 5; export let

[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...

GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/858 METRON-1344: Externalize the infrastructural components using integration tests ## Contributor Comments This PR will add infrastructure to our Travis build that will allow the Alerts UI e2e

Re: Heterogeneous indexing batch size for different Metron feeds

Sorry, We flush for timeouts on every storm ‘tick’ message, not on every message. On December 6, 2017 at 08:29:51, Otto Fowler (ottobackwa...@gmail.com) wrote: I have looked at it. We maintain batch lists for each sensor which gather messages to index. When we get a message that puts it over

Re: Heterogeneous indexing batch size for different Metron feeds

I have looked at it. We maintain batch lists for each sensor which gather messages to index. When we get a message that puts it over the batch size the messages are flushed and written to the target. There is also a timeout component, where the batch would be flushed based on timeout. While

Re: Heterogeneous indexing batch size for different Metron feeds

Everything looks normal except the high number of failed tuples. Do you know how the indexing batch size works? Based on our observations it seems it doesn't update the messages that are in enrichments and indexing topics. On Thu, Dec 7, 2017 at 12:13 AM, Otto Fowler

Re: Heterogeneous indexing batch size for different Metron feeds

What do you see in the storm ui for the indexing topology? On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com) wrote: Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It mpacts topology error rate and cause almost 90% error rate on indexing tuples. On 6

[GitHub] metron pull request #857: METRON-1340: Improve e2e tests for metron alerts

Github user iraghumitra commented on a diff in the pull request: https://github.com/apache/metron/pull/857#discussion_r155219994 --- Diff: metron-interface/metron-alerts/src/app/utils/constants.ts --- @@ -38,5 +38,6 @@ export let TREE_SUB_GROUP_SIZE = 5; export let

Re: Heterogeneous indexing batch size for different Metron feeds

Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It mpacts topology error rate and cause almost 90% error rate on indexing tuples. On 6 Dec. 2017 00:20, "Otto Fowler" wrote: Where are you seeing the errors? Screenshot? On December 5, 2017 at

[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

Github user iraghumitra commented on the issue: https://github.com/apache/metron/pull/857 @merrimanr - I removed the node version check in the ui code - The browser.sleep is used in following areas - When we are using methods like waitForText, few times I see the