Re: [DISCUSS] Error Indexing

Simply as a unique identifier of the original information which is failing some step, and thus giving you something to key in on and create a count of unique events and prioritize issues without the concern of cyclical issues (if the issue is with indexing a specific message, and you try to index

Re: [Discuss] Improve Alerting

Otto, I think you're thinking of the "Enrich enrichment" dev mailing list thread that Dima started. In that case we chatted about passing through enrichment multiple times while decrementing a TTL field to prevent infinite loops (and drop a message to the error queue if TTL == 0) would work just

[GitHub] incubator-metron issue #316: METRON-503: Metron REST API

Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr you are right about guava. I'll remove my uses of it. Plus, Java has a lot of those functions built in now. No reason I shouldn't use those instead. I agree that the

Re: [DISCUSS] Metron Management UI

Ok great, thanks for the feedback Ryan. I'm going to try and get around to playing with this next week if I can. Currently my productivity machine is "in the shop" getting the battery replaced, so I'm playing man down right now. So if we ignore the current API restrictions, here are my MUST

Re: [DISCUSS] Metron Management UI

The MPack creates elastic templates for the standard parsers included in the pack. However, when creating new sensors we should be able to infer Elastic templates from the schema. There are certain fields that we need to mark types for to generate those templates (e.g. date / time fields).

Re: [DISCUSS] Metron Management UI

The Index templates are currently handled in the Ambari MPack (unless you are suggesting some additional functionality beyond what the Ambari MPack gives us today.) I think there is always going to be a fuzzy, gray line between what the Ambari MPack should do and what the Management UI should do.

Re: [Discuss] Improve Alerting

Isn’t this similar to the discussion around multi-pass enrichment? JonZ -> you are good at looking up discussions ;) On February 1, 2017 at 14:38:05, Casey Stella (ceste...@gmail.com) wrote: Yeah, I think your solution and mine are the same based on reading your suggestion. Just add a write

Re: [Discuss] Improve Alerting

Agreed. What do you think about using the existing Indexing topology to write the data to HBase for the profiler? - The Profiler would have only one output; Kafka. The Profiler would not write to HBase. - Since the Profiler is just another source of telemetry, it is parsed,

Re: [Discuss] Improve Alerting

Yeah, I think your solution and mine are the same based on reading your suggestion. Just add a write section to the profile and you can write right back into the kafka queue and get all the triage goodness. You would need to ensure that you don't end up with infinite loops back in the profiler.

Re: [Discuss] Improve Alerting

I would be open to doing it this way. It makes the implementation simpelr. The only problem with that is if I want to just use a string, I would have to embed a quote. "comment": " 'This is my rule comment with no values'" Not really a big deal though. On Wed, Feb 1, 2017 at 2:28 PM, Casey

Re: [Discuss] Improve Alerting

Great. I think we're thinking along the same lines. I just sent a follow-up of another proposal that takes this idea a little further. What if we treated the Profiler as another source of telemetry? On Wed, Feb 1, 2017 at 2:23 PM, Casey Stella wrote: > Regarding point 2,

Re: [Discuss] Improve Alerting

*Problem* Triage Calculated Values from the Profiler The value being interrogated here, the number of inbound flows, is not a static value contained within any single telemetry message. This value is calculated across multiple messages by the Profiler. The current Threat Triage process cannot

Re: [Discuss] Improve Alerting

I like the direction. One thing that we may want is for comment to just be a stellar expression and construct a function to essentially do String.format(). So, that'd become: "triageConfig" : { "riskLevelRules" : [ { "name" : "Abnormal Value", "comment" : "FORMAT('For %s; the

Re: [Discuss] Improve Alerting

Regarding point 2, could we enable the profiler to write data to kafka and the enrichment queue? I'm proposing the profiler do something like this: - Count the number of inbound flows - On the tick, send a message to the enrichment queue containing: - the number of flows - A

Re: [Discuss] Improve Alerting

Like I said, here is a proposed solution to one of the gaps I identified in the previous email. *Problem* There is little transparency into the Threat Triage process itself. When Threat Triage runs, all I get is a score. I don't know how that score was arrived at, which rules were triggered,

Re: [DISCUSS] Metron Management UI

Hello Dima, Thank you for taking a few minutes to look at this new addition. I hope that it can make working with Metron much easier. Do you think ElasticSearch Index Templates are a MUST have initially? Or is it okay to start here? I do like the idea of adding that functionality. Houshang

[Discuss] Improve Alerting

I'd like to explore the functionality that we have in Metron using a motivating example. I think this will help highlight some gaps where we can enhance Metron. The motivating example is that I would like to create an alert if the number of inbound flows to any host over a 15 minute interval is

[GitHub] incubator-metron issue #430: METRON-658: Updated Grammar to Handle More Uses...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/430 +1 by inspection, good contribution. I like the refactoring of the grammar too. Stellar is getting cleaner and cleaner with every contribution like this. --- If your project is set up

[GitHub] incubator-metron pull request #426: METRON-675: Make Threat Triage rules abl...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/426 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] incubator-metron issue #426: METRON-675: Make Threat Triage rules able to be...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/426 @nickwallen Agreed on the aggregator being stellar. I created https://issues.apache.org/jira/browse/METRON-683 for it --- If your project is set up for it, you can reply to this email

[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/432 # Testing Plan ## Preliminaries * Download the alexa 1m dataset: ``` wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip unzip top-1m.csv.zip ```

[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/432 I know it seems like a lot of code changed, but a lot of this was reorganizing the flat file loader class and splitting it into separate reusable components, rather than new code. ---

[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...

GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/432 METRON-682: Unify and Improve the Flat File Loader Currently the flat file loader is deficient in a couple ways: * It only supports importing local data despite there being a

Re: [DISCUSS] Metron Management UI

This looks awesome! A great starting point. Like Jon, I'm looking forward to kicking the tires. -Kyle On Wed, Feb 1, 2017 at 10:18 AM, Ryan Merriman wrote: > Jon, I've done my best to keep this UI in sync with the API PR so you can > play around with it now if you want.

[DISCUSS] Metron GUI Syleguide

I would like to start a discussion around what would be a good style guide for a angular/SASS/bootstrap based GUI for metron. A coding styleguide (note, not a visual styleguide) is a valuable tool for teams that strives for a degree of standardization in their code. A good styleguide, when

Re: [DISCUSS] Metron Management UI

Thank you Houshang, That looks absolutely great, loved it! Will it make sense to also add ElasticSearch index templates querying/creation here? - Dima On 02/01/2017 02:06 AM, Houshang Livian wrote: > Hello Metron Community, > > We have constructed a Management Module UI, built on top of

[GitHub] incubator-metron pull request #431: METRON-680: GeoLiteDatabase incorrectly ...

Github user justinleet closed the pull request at: https://github.com/apache/incubator-metron/pull/431 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

Re: [DISCUSS] Error Indexing

That's a great topic of discussion. Throughout the thread the idea of having hash of the message that failed is changed, can someone please explain why do you plan to use this hash and how? - Dima On 02/01/2017 06:23 AM, zeo...@gmail.com wrote: > After thinking on this for a few days I recant