Re: [DISCUSS] Error Indexing

[zeo...@gmail.com]

Simply as a unique identifier of the original information which is failing
some step, and thus giving you something to key in on and create a count of
unique events and prioritize issues without the concern of cyclical issues
(if the issue is with indexing a specific message, and you try to index it
again, it will just fail in a loop).

Jon

On Wed, Feb 1, 2017 at 6:59 AM Dima Kovalyov 
wrote:

> That's a great topic of discussion.
>
> Throughout the thread the idea of having hash of the message that failed
> is changed, can someone please explain why do you plan to use this hash
> and how?
>
> - Dima
>
> On 02/01/2017 06:23 AM, zeo...@gmail.com wrote:
> > After thinking on this for a few days I recant my previous suggestion of
> > TupleHash256.  It's still a bit early for SHA-3 - no good reference
> > implementations/libraries exist (I did some searching and emailing), it
> is
> > optimized for hardware but no hardware implementation is widely
> accessible,
> > FIPS 140-3 is still not close to finalized, etc.
> >
> > I think we could simulate the benefits of tuplehash by sorting the
> tuples,
> > then doing SHA-256(len(tuple1) | tuple1 | ... | len(tuplen) | tuplen).
> > Happy to entertain opposing thoughts, such as BLAKE2, etc. but with the
> > likely users of Metron, I think sticking with FIPS 140-2 is a solid
> choice.
> >
> > Jon
> >
> > On Thu, Jan 26, 2017, 11:23 AM zeo...@gmail.com 
> wrote:
> >
> > So one more thing regarding why I think we should throw an exception on a
> > failed enrichment.  If we do make something like username a constant
> field,
> > in cases where that is used to calculate rawMessage_hash, if it fails to
> > enrich, the hash would be different compared to when it succeeds.  Of
> > course I think the initial intent of adding username as a constant field
> > would be to handle it in the parsers, where that information is provided
> in
> > the messages themselves, but how would Threat Intel know the difference?
> > In my environment I am looking forward to a streaming enrichment that
> adds
> > the username, where applicable, anywhere I have an IP.
> >
> > My hesitant suggestion for a hashing algorithm would be to use
> > TupleHash256, as it is a NIST-provided implementation of SHA-3 (using
> > cSHAKE) for this use case.  Details here
> > <
> http://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-185.pdf>.
> > However, I haven't been able to find a reference implementation of this
> in
> > any language, so that's a bit of a downside.  A more general SHA3-256
> > implementation where we handle ordering could work as well, but would be
> > significantly less optimal.
> >
> > Jon
> >
> > On Thu, Jan 26, 2017 at 10:20 AM Ryan Merriman 
> wrote:
> >
> > Jon, I misread the code in the GenericEnrichmentBolt.  The error is
> > forwarded on so no issues there.
> >
> > Defaulting to the common fields makes sense.  I will dig into the
> > GenericEnrichmentBolt more, maybe there is a way to get the error fields
> > without having to significantly change things.  Any opinion on a hashing
> > algorithm?
> >
> > On Wed, Jan 25, 2017 at 9:37 PM, zeo...@gmail.com 
> wrote:
> >
> >> Although hashing the whole message is better than nothing, it misses a
> lot
> >> of the benefits we could get.
> >>
> >> While I'd love to have consistency for this field across all of the
> >> different error.types, it appears that may not be reasonably possible
> >> because of the parsers.  So, how about something like hash all of the
> >> constant
> >> fields
> >>  >> metron-platform/metron-common/src/main/java/org/apache/
> >> metron/common/Constants.java>
> >> excluding
> >> timestamp and original_string unless it is a parser, in which case hash
> > the
> >> entire message?  This gives us some measure of event uniqueness and it
> can
> >> grow as we define additional constant fields (I recall discussing with
> >> someone else on the list regarding expanding those standard fields to
> >> include things like usernames but I can't find the specific email
> >> exchange).
> >>
> >> Because some enrichments can be heavily relied on, I think it makes
> sense
> >> to put a message onto the error queue when it throws an exception.  Not
> >> only does this help troubleshoot edge cases, but it makes issues more
> >> obvious when assembling a new enrichment in dev/test.  I can't think of
> a
> >> scenario currently where an enrichment would only be "best effort" and
> > that
> >> I wouldn't want that error indexed and retrievable.  However, this gets
> >> interesting when talking about the various options to solve the "Enrich
> >> enrichment" discussion from earlier in the month.  We can keep that part
> > of
> >> this separate though, as I don't think that's being actively pursued
> right
> >> now.
> >>
> >> Jon
> >>
> >> On Wed, Jan 25, 2017 at 10:49 AM David Lyle 
> wrote:
> >>
> >> RE: separate JIRA for MPack/Ansible. No objection to tracking them
> >> 

Re: [Discuss] Improve Alerting

[zeo...@gmail.com]

Otto, I think you're thinking of the "Enrich enrichment" dev mailing list
thread that Dima started.  In that case we chatted about passing through
enrichment multiple times while decrementing a TTL field to prevent
infinite loops (and drop a message to the error queue if TTL == 0) would
work just fine, and that sounds like it may relate here as well.

I'm not comfortable enough with the profiler in this scenario to comment
authoritatively, other than I can say that I'm really liking the
suggestions that have gone back and forth so far.

Jon

On Wed, Feb 1, 2017 at 2:48 PM Otto Fowler  wrote:

Isn’t this similar to the discussion around multi-pass enrichment?

JonZ -> you are good at looking up discussions ;)



On February 1, 2017 at 14:38:05, Casey Stella (ceste...@gmail.com) wrote:

Yeah, I think your solution and mine are the same based on reading your
suggestion. Just add a write section to the profile and you can write
right back into the kafka queue and get all the triage goodness. You would
need to ensure that you don't end up with infinite loops back in the
profiler. So, things like profiles that interact with EVERY message and
send a message back to the kafka queue in enrichment would be bad.

On Wed, Feb 1, 2017 at 2:35 PM, Nick Allen  wrote:

> Great. I think we're thinking along the same lines. I just sent a
> follow-up of another proposal that takes this idea a little further. What
> if we treated the Profiler as another source of telemetry?
>
> On Wed, Feb 1, 2017 at 2:23 PM, Casey Stella  wrote:
>
> > Regarding point 2, could we enable the profiler to write data to kafka
> and
> > the enrichment queue?
> >
> > I'm proposing the profiler do something like this:
> >
> > - Count the number of inbound flows
> > - On the tick, send a message to the enrichment queue containing:
> > - the number of flows
> > - A source type of 'system_alert'
> > - is_alert set to true
> > - In enrichment, we enrich and triage system_alert source data in the
> > same way we do any other.
> >
> > This would not solve the transparency issue, but at least make it so we
> > keep triage in one place in the architecture. Also, enabling kafka
> writing
> > would enable other types of use-cases, like situations where we find
> > outliers *directly* in the profile and send the alerts directly to the
> > indexing queue without triage.
> >
> > The only changes this proposal would require would be
> >
> > 1. a "write" section to a profile that takes a list of stellar
> > statements and gets run on the tick write
> > 2. fixing the kafka writing stellar functions
> >
> > Casey
> >
> > On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:
> >
> > > I'd like to explore the functionality that we have in Metron using a
> > > motivating example. I think this will help highlight some gaps where
> we
> > > can enhance Metron.
> > >
> > > The motivating example is that I would like to create an alert if the
> > > number of inbound flows to any host over a 15 minute interval is
> > abnormal.
> > > I would like the alert to contain the specific information below to
> > > streamline the triage process.
> > >
> > > Rule: Abnormal number of inbound flows
> > > Bin: 15 mins
> > > Alert: The host 'powned.svr.bank.com' has '230' inbound flows,
> exceeding
> > > the threshold of '202'
> > >
> > >
> > > *What Works*
> > >
> > > In some ways, this example is similar to the "Outlier Detection" demo
> > that
> > > I performed with the Profiler a few months back. We have most of what
> > we
> > > need to do this with a couple caveats.
> > >
> > > 1. An enrichment would be added to enrich the message with the
correct
> > > internal hostname 'powned.svr.bank.com'.
> > >
> > > 2. With the Profiler, I can capture some idea of what "normal" is for
> the
> > > number of inbound flows across 15 minute intervals.
> > > 3. With Threat Triage, I can create rules that alert when a value
> exceeds
> > > what the Profiler defines as normal.
> > >
> > >
> > > *What's Missing*
> > >
> > > Its nice to know that we are almost all the way there with this
> example.
> > > Unfortunately, there are two gaps that fall out of this.
> > >
> > > 1. *Threat Triage Transparency*
> > >
> > > There is little transparency into the Threat Triage process itself.
> When
> > > Threat Triage runs, all I get is a score. I don't know how that score
> > was
> > > arrived at, which rules were triggered, and the specific values that
> > caused
> > > a rule to trigger.
> > >
> > > More specifically, there is no way to generate a message that looks
> like
> > > "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
the
> > > threshold of '202'".
> > >
> > >
> > > 2. *Triage Calculated Values from the Profiler*
> > >
> > > Also, the value being interrogated here, the number of inbound flows,
> is
> > > not a static value contained within any single telemetry message.
This
> > > value is calculated across multiple messages by the Profiler. The
> > current
> > > Threat Triage process ca

[GitHub] incubator-metron issue #316: METRON-503: Metron REST API

[jjmeyer0]

Github user jjmeyer0 commented on the issue:

https://github.com/apache/incubator-metron/pull/316
  
@merrimanr you are right about guava. I'll remove my uses of it. Plus, Java 
has a lot of those functions built in now. No reason I shouldn't use those 
instead.

I agree that the guava issue should be taken care of separately. Right now, 
I think we should allow a user to set an envitonment variable to point to which 
guava to use (similar to hibernate and mysql). This will make it easier for us 
to run it outside of our IDE.

I think your changes look pretty good. Really nice clean up/tests. We've 
changed a lot, so I want to do another pass though.

I think we should still have our `HdfsService`. Right now, I think using 
`FileSystem` directly works for us. Using Knox is a good idea, but it probably 
needs to be optional. My thought is having the service will help us achieve 
that. You are probably right though. If we can achieve everything with WebHDFS 
it's better for our service to use that. This PR is already really big though. 
Maybe we can have that as a take away?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Metron Management UI

[zeo...@gmail.com]

Ok great, thanks for the feedback Ryan.  I'm going to try and get around to
playing with this next week if I can.  Currently my productivity machine is
"in the shop" getting the battery replaced, so I'm playing man down right
now.

So if we ignore the current API restrictions, here are my MUST haves:

 - We should be able to update search/retrieve (i.e. I agree with Dima).
This means tweaking the ES templates, because when putting together a new
sensor or modifying an existing one, we will likely have a new field or
entirely new template.
 - When creating a new sensor, it should create a new kafka topic (if it
doesn't already - it may, I just haven't played with it yet) with some very
simple/dumb default settings (1 partition, default topic size, 0
replication).

And here are my NICE to haves (eventually):
 - You should have the ability to tune a kafka topic's config from the UI,
such as setting the topic size, # of partitions, and replication number.
This is especially useful if you just created the kafka topic in the UI.
Of course this should have lots of warnings about the issues with
decreasing these numbers in the future, and maybe a "read more" link to
some related documentation in the kafka documentation.

 - Tune and restart the topologies to add parallelism into storm (things
like the --parser_p, --parser_num_tasks, --num_workers, etc. options).
 - It would be nice to be able to modify the jobs that purge data from HDFS
and ES via the UI.  Managing the cluster should include setting your data
retention wants - I think these are still purge cron jobs?  For some
context, I'm thinking about how this could manage METRON-477 (both the
original ticket as well as Nick's comments for additional features), which
could be managed in this UI.
 - The ability to generate a mpack that would install the currently
configured state of the cluster (not including data, of course, but
focusing on setting new, tweaked defaults to be what is currently
configured).  This would be useful for rebuilds, standing up new clusters,
or migrating to a different environment.  IMO, this is a really slick
feature because it allows a nice migration from dev > test > prod.


Not to get off topic, but I think the ES side of the house could generally
use some love.  For instance, I don't see a reason why every string field
shouldn't have '"ignore_above": 8191' on it, at least right now.  I will
probably do this as a part of METRON-635, but we should make sure to carry
that setting forward appropriately (i.e. if we infer ES templates from a
schema).  I could agree with ES templates being inferred when creating new
parsers/sensors as a nice to have, building off of a must have of being
able to make/tweak ES templates in the UI/API.


Ok, that's a lot of text, I'll end it there.  =D


Jon

On Wed, Feb 1, 2017 at 3:07 PM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

The MPack creates elastic templates for the standard parsers included in
the pack.

However, when creating new sensors we should be able to infer Elastic
templates from the schema. There are certain fields that we need to mark
types for to generate those templates (e.g. date / time fields). This goes
back to the debate we had a while ago about, for example, typing in the
grok parser, but applies for others too. I suspect that since we should be
able to infer all the types needed for templates, this is more an excellent
feature enhancement for the REST api layer, unless anyone can think of a
reason we would need explicit type definition in the schema config element
of the UI. Thoughts?

Simon


> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
>
> The Index templates are currently handled in the Ambari MPack (unless you
> are suggesting some additional functionality beyond what the Ambari MPack
> gives us today.)
>
> I think there is always going to be a fuzzy, gray line between what the
> Ambari MPack should do and what the Management UI should do.  I think it
> works fine in Ambari today, but I could be convinced otherwise.
>
>
>
> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov 
> wrote:
>
>> Thank you Houshang,
>>
>> That looks absolutely great, loved it!
>> Will it make sense to also add ElasticSearch index  templates
>> querying/creation here?
>>
>> - Dima
>>
>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
>>> Hello Metron Community,
>>>
>>> We have constructed a Management Module UI, built on top of METRON-503
>> (REST API) (currently under review). This Module gives users the ability
to
>> setup and administer much of the product through the UI.
>>>
>>> Here are some screens to show you what we are thinking. Please take a
>> look:
>>> http://imgur.com/a/QAQyu?
>>>
>>> Does this look like a reasonable place to start?
>>> Is there anything that is an absolute MUST have or MUST NOT have?
>>>
>>> Houshang Livian
>>> Senior User Experience Designer
>>> Hortonworks
>>>
>>> www.hortonworks.com
>>> 
>>>
>>> M

Re: [DISCUSS] Metron Management UI

[Simon Elliston Ball]

The MPack creates elastic templates for the standard parsers included in the 
pack. 

However, when creating new sensors we should be able to infer Elastic templates 
from the schema. There are certain fields that we need to mark types for to 
generate those templates (e.g. date / time fields). This goes back to the 
debate we had a while ago about, for example, typing in the grok parser, but 
applies for others too. I suspect that since we should be able to infer all the 
types needed for templates, this is more an excellent feature enhancement for 
the REST api layer, unless anyone can think of a reason we would need explicit 
type definition in the schema config element of the UI. Thoughts? 

Simon


> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
> 
> The Index templates are currently handled in the Ambari MPack (unless you
> are suggesting some additional functionality beyond what the Ambari MPack
> gives us today.)
> 
> I think there is always going to be a fuzzy, gray line between what the
> Ambari MPack should do and what the Management UI should do.  I think it
> works fine in Ambari today, but I could be convinced otherwise.
> 
> 
> 
> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov 
> wrote:
> 
>> Thank you Houshang,
>> 
>> That looks absolutely great, loved it!
>> Will it make sense to also add ElasticSearch index  templates
>> querying/creation here?
>> 
>> - Dima
>> 
>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
>>> Hello Metron Community,
>>> 
>>> We have constructed a Management Module UI, built on top of METRON-503
>> (REST API) (currently under review). This Module gives users the ability to
>> setup and administer much of the product through the UI.
>>> 
>>> Here are some screens to show you what we are thinking. Please take a
>> look:
>>> http://imgur.com/a/QAQyu?
>>> 
>>> Does this look like a reasonable place to start?
>>> Is there anything that is an absolute MUST have or MUST NOT have?
>>> 
>>> Houshang Livian
>>> Senior User Experience Designer
>>> Hortonworks
>>> 
>>> www.hortonworks.com
>>> 
>>> 
>>> Mobile: (831) 521-4176
>>> hliv...@hortonworks.com
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> -- 
> Nick Allen 

Re: [DISCUSS] Metron Management UI

[Nick Allen]

The Index templates are currently handled in the Ambari MPack (unless you
are suggesting some additional functionality beyond what the Ambari MPack
gives us today.)

I think there is always going to be a fuzzy, gray line between what the
Ambari MPack should do and what the Management UI should do.  I think it
works fine in Ambari today, but I could be convinced otherwise.



On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov 
wrote:

> Thank you Houshang,
>
> That looks absolutely great, loved it!
> Will it make sense to also add ElasticSearch index  templates
> querying/creation here?
>
> - Dima
>
> On 02/01/2017 02:06 AM, Houshang Livian wrote:
> > Hello Metron Community,
> >
> > We have constructed a Management Module UI, built on top of METRON-503
> (REST API) (currently under review). This Module gives users the ability to
> setup and administer much of the product through the UI.
> >
> > Here are some screens to show you what we are thinking. Please take a
> look:
> > http://imgur.com/a/QAQyu?
> >
> > Does this look like a reasonable place to start?
> > Is there anything that is an absolute MUST have or MUST NOT have?
> >
> > Houshang Livian
> > Senior User Experience Designer
> > Hortonworks
> >
> > www.hortonworks.com
> > 
> >
> > Mobile: (831) 521-4176
> > hliv...@hortonworks.com
> > 
> >
> >
>
>


-- 
Nick Allen 

Re: [Discuss] Improve Alerting

[Otto Fowler]

Isn’t this similar to the discussion around multi-pass enrichment?

JonZ -> you are good at looking up discussions ;)



On February 1, 2017 at 14:38:05, Casey Stella (ceste...@gmail.com) wrote:

Yeah, I think your solution and mine are the same based on reading your
suggestion. Just add a write section to the profile and you can write
right back into the kafka queue and get all the triage goodness. You would
need to ensure that you don't end up with infinite loops back in the
profiler. So, things like profiles that interact with EVERY message and
send a message back to the kafka queue in enrichment would be bad.

On Wed, Feb 1, 2017 at 2:35 PM, Nick Allen  wrote:

> Great. I think we're thinking along the same lines. I just sent a
> follow-up of another proposal that takes this idea a little further. What
> if we treated the Profiler as another source of telemetry?
>
> On Wed, Feb 1, 2017 at 2:23 PM, Casey Stella  wrote:
>
> > Regarding point 2, could we enable the profiler to write data to kafka
> and
> > the enrichment queue?
> >
> > I'm proposing the profiler do something like this:
> >
> > - Count the number of inbound flows
> > - On the tick, send a message to the enrichment queue containing:
> > - the number of flows
> > - A source type of 'system_alert'
> > - is_alert set to true
> > - In enrichment, we enrich and triage system_alert source data in the
> > same way we do any other.
> >
> > This would not solve the transparency issue, but at least make it so we
> > keep triage in one place in the architecture. Also, enabling kafka
> writing
> > would enable other types of use-cases, like situations where we find
> > outliers *directly* in the profile and send the alerts directly to the
> > indexing queue without triage.
> >
> > The only changes this proposal would require would be
> >
> > 1. a "write" section to a profile that takes a list of stellar
> > statements and gets run on the tick write
> > 2. fixing the kafka writing stellar functions
> >
> > Casey
> >
> > On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:
> >
> > > I'd like to explore the functionality that we have in Metron using a
> > > motivating example. I think this will help highlight some gaps where
> we
> > > can enhance Metron.
> > >
> > > The motivating example is that I would like to create an alert if the
> > > number of inbound flows to any host over a 15 minute interval is
> > abnormal.
> > > I would like the alert to contain the specific information below to
> > > streamline the triage process.
> > >
> > > Rule: Abnormal number of inbound flows
> > > Bin: 15 mins
> > > Alert: The host 'powned.svr.bank.com' has '230' inbound flows,
> exceeding
> > > the threshold of '202'
> > >
> > >
> > > *What Works*
> > >
> > > In some ways, this example is similar to the "Outlier Detection" demo
> > that
> > > I performed with the Profiler a few months back. We have most of what
> > we
> > > need to do this with a couple caveats.
> > >
> > > 1. An enrichment would be added to enrich the message with the
correct
> > > internal hostname 'powned.svr.bank.com'.
> > >
> > > 2. With the Profiler, I can capture some idea of what "normal" is for
> the
> > > number of inbound flows across 15 minute intervals.
> > > 3. With Threat Triage, I can create rules that alert when a value
> exceeds
> > > what the Profiler defines as normal.
> > >
> > >
> > > *What's Missing*
> > >
> > > Its nice to know that we are almost all the way there with this
> example.
> > > Unfortunately, there are two gaps that fall out of this.
> > >
> > > 1. *Threat Triage Transparency*
> > >
> > > There is little transparency into the Threat Triage process itself.
> When
> > > Threat Triage runs, all I get is a score. I don't know how that score
> > was
> > > arrived at, which rules were triggered, and the specific values that
> > caused
> > > a rule to trigger.
> > >
> > > More specifically, there is no way to generate a message that looks
> like
> > > "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
the
> > > threshold of '202'".
> > >
> > >
> > > 2. *Triage Calculated Values from the Profiler*
> > >
> > > Also, the value being interrogated here, the number of inbound flows,
> is
> > > not a static value contained within any single telemetry message.
This
> > > value is calculated across multiple messages by the Profiler. The
> > current
> > > Threat Triage process cannot be used to interrogate values calculated
> by
> > > the Profiler.
> > >
> > >
> > > To try and keep this email concise and digestible, I am going to send
a
> > > follow-on discussing proposed solutions for each of these separately.
> > >
> >
>
>
>
> --
> Nick Allen 
>

Re: [Discuss] Improve Alerting

[Nick Allen]

Agreed.

What do you think about using the existing Indexing topology to write the
data to HBase for the profiler?

   - The Profiler would have only one output; Kafka.  The Profiler would
   not write to HBase.


   - Since the Profiler is just another source of telemetry, it is parsed,
   enriched, triaged, and then indexed.


   - Thanks to your recent work we can now configure each 'indexer'
   separately, so we would just have an HBase indexer.

Seems like a logical extension of this idea.  Probably a little
overreaching as a first pass.  Maybe that is something we can evolve
towards.



On Wed, Feb 1, 2017 at 2:37 PM, Casey Stella  wrote:

> Yeah, I think your solution and mine are the same based on reading your
> suggestion.  Just add a write section to the profile and you can write
> right back into the kafka queue and get all the triage goodness.  You would
> need to ensure that you don't end up with infinite loops back in the
> profiler.  So, things like profiles that interact with EVERY message and
> send a message back to the kafka queue in enrichment would be bad.
>
> On Wed, Feb 1, 2017 at 2:35 PM, Nick Allen  wrote:
>
> > Great.  I think we're thinking along the same lines.  I just sent a
> > follow-up of another proposal that takes this idea a little further.
> What
> > if we treated the Profiler as another source of telemetry?
> >
> > On Wed, Feb 1, 2017 at 2:23 PM, Casey Stella  wrote:
> >
> > > Regarding point 2, could we enable the profiler to write data to kafka
> > and
> > > the enrichment queue?
> > >
> > > I'm proposing the profiler do something like this:
> > >
> > >- Count the number of inbound flows
> > >- On the tick, send a message to the enrichment queue containing:
> > >   - the number of flows
> > >   - A source type of 'system_alert'
> > >   - is_alert set to true
> > >- In enrichment, we enrich and triage system_alert source data in
> the
> > >same way we do any other.
> > >
> > > This would not solve the transparency issue, but at least make it so we
> > > keep triage in one place in the architecture.  Also, enabling kafka
> > writing
> > > would enable other types of use-cases, like situations where we find
> > > outliers *directly* in the profile and send the alerts directly to the
> > > indexing queue without triage.
> > >
> > > The only changes this proposal would require would be
> > >
> > >1. a "write" section to a profile that takes a list of stellar
> > >statements and gets run on the tick write
> > >2. fixing the kafka writing stellar functions
> > >
> > > Casey
> > >
> > > On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:
> > >
> > > > I'd like to explore the functionality that we have in Metron using a
> > > > motivating example.  I think this will help highlight some gaps where
> > we
> > > > can enhance Metron.
> > > >
> > > > The motivating example is that I would like to create an alert if the
> > > > number of inbound flows to any host over a 15 minute interval is
> > > abnormal.
> > > > I would like the alert to contain the specific information below to
> > > > streamline the triage process.
> > > >
> > > > Rule: Abnormal number of inbound flows
> > > > Bin: 15 mins
> > > > Alert: The host 'powned.svr.bank.com' has '230' inbound flows,
> > exceeding
> > > > the threshold of '202'
> > > >
> > > >
> > > > *What Works*
> > > >
> > > > In some ways, this example is similar to the "Outlier Detection" demo
> > > that
> > > > I performed with the Profiler a few months back.   We have most of
> what
> > > we
> > > > need to do this with a couple caveats.
> > > >
> > > > 1. An enrichment would be added to enrich the message with the
> correct
> > > > internal hostname 'powned.svr.bank.com'.
> > > >
> > > > 2. With the Profiler, I can capture some idea of what "normal" is for
> > the
> > > > number of inbound flows across 15 minute intervals.
> > > > 3. With Threat Triage, I can create rules that alert when a value
> > exceeds
> > > > what the Profiler defines as normal.
> > > >
> > > >
> > > > *What's Missing*
> > > >
> > > > Its nice to know that we are almost all the way there with this
> > example.
> > > > Unfortunately, there are two gaps that fall out of this.
> > > >
> > > >  1. *Threat Triage Transparency*
> > > >
> > > > There is little transparency into the Threat Triage process itself.
> > When
> > > > Threat Triage runs, all I get is a score.  I don't know how that
> score
> > > was
> > > > arrived at, which rules were triggered, and the specific values that
> > > caused
> > > > a rule to trigger.
> > > >
> > > > More specifically, there is no way to generate a message that looks
> > like
> > > > "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
> the
> > > > threshold of '202'".
> > > >
> > > >
> > > > 2. *Triage Calculated Values from the Profiler*
> > > >
> > > > Also, the value being interrogated here, the number of inbound flows,
> > is
> > > > not a static value contained within any single 

Re: [Discuss] Improve Alerting

[Casey Stella]

Yeah, I think your solution and mine are the same based on reading your
suggestion.  Just add a write section to the profile and you can write
right back into the kafka queue and get all the triage goodness.  You would
need to ensure that you don't end up with infinite loops back in the
profiler.  So, things like profiles that interact with EVERY message and
send a message back to the kafka queue in enrichment would be bad.

On Wed, Feb 1, 2017 at 2:35 PM, Nick Allen  wrote:

> Great.  I think we're thinking along the same lines.  I just sent a
> follow-up of another proposal that takes this idea a little further.  What
> if we treated the Profiler as another source of telemetry?
>
> On Wed, Feb 1, 2017 at 2:23 PM, Casey Stella  wrote:
>
> > Regarding point 2, could we enable the profiler to write data to kafka
> and
> > the enrichment queue?
> >
> > I'm proposing the profiler do something like this:
> >
> >- Count the number of inbound flows
> >- On the tick, send a message to the enrichment queue containing:
> >   - the number of flows
> >   - A source type of 'system_alert'
> >   - is_alert set to true
> >- In enrichment, we enrich and triage system_alert source data in the
> >same way we do any other.
> >
> > This would not solve the transparency issue, but at least make it so we
> > keep triage in one place in the architecture.  Also, enabling kafka
> writing
> > would enable other types of use-cases, like situations where we find
> > outliers *directly* in the profile and send the alerts directly to the
> > indexing queue without triage.
> >
> > The only changes this proposal would require would be
> >
> >1. a "write" section to a profile that takes a list of stellar
> >statements and gets run on the tick write
> >2. fixing the kafka writing stellar functions
> >
> > Casey
> >
> > On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:
> >
> > > I'd like to explore the functionality that we have in Metron using a
> > > motivating example.  I think this will help highlight some gaps where
> we
> > > can enhance Metron.
> > >
> > > The motivating example is that I would like to create an alert if the
> > > number of inbound flows to any host over a 15 minute interval is
> > abnormal.
> > > I would like the alert to contain the specific information below to
> > > streamline the triage process.
> > >
> > > Rule: Abnormal number of inbound flows
> > > Bin: 15 mins
> > > Alert: The host 'powned.svr.bank.com' has '230' inbound flows,
> exceeding
> > > the threshold of '202'
> > >
> > >
> > > *What Works*
> > >
> > > In some ways, this example is similar to the "Outlier Detection" demo
> > that
> > > I performed with the Profiler a few months back.   We have most of what
> > we
> > > need to do this with a couple caveats.
> > >
> > > 1. An enrichment would be added to enrich the message with the correct
> > > internal hostname 'powned.svr.bank.com'.
> > >
> > > 2. With the Profiler, I can capture some idea of what "normal" is for
> the
> > > number of inbound flows across 15 minute intervals.
> > > 3. With Threat Triage, I can create rules that alert when a value
> exceeds
> > > what the Profiler defines as normal.
> > >
> > >
> > > *What's Missing*
> > >
> > > Its nice to know that we are almost all the way there with this
> example.
> > > Unfortunately, there are two gaps that fall out of this.
> > >
> > >  1. *Threat Triage Transparency*
> > >
> > > There is little transparency into the Threat Triage process itself.
> When
> > > Threat Triage runs, all I get is a score.  I don't know how that score
> > was
> > > arrived at, which rules were triggered, and the specific values that
> > caused
> > > a rule to trigger.
> > >
> > > More specifically, there is no way to generate a message that looks
> like
> > > "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
> > > threshold of '202'".
> > >
> > >
> > > 2. *Triage Calculated Values from the Profiler*
> > >
> > > Also, the value being interrogated here, the number of inbound flows,
> is
> > > not a static value contained within any single telemetry message.  This
> > > value is calculated across multiple messages by the Profiler.  The
> > current
> > > Threat Triage process cannot be used to interrogate values calculated
> by
> > > the Profiler.
> > >
> > >
> > > To try and keep this email concise and digestible, I am going to send a
> > > follow-on discussing proposed solutions for each of these separately.
> > >
> >
>
>
>
> --
> Nick Allen 
>

Re: [Discuss] Improve Alerting

[Nick Allen]

I would be open to doing it this way.  It makes the implementation simpelr.

The only problem with that is if I want to just use a string, I would have
to embed a quote.

"comment": " 'This is my rule comment with no values'"


Not really a big deal though.



On Wed, Feb 1, 2017 at 2:28 PM, Casey Stella  wrote:

> I like the direction.  One thing that we may want is for comment to just be
> a stellar expression and construct a function to essentially do
> String.format().  So, that'd become:
> "triageConfig" : {
>   "riskLevelRules" : [
> {
>   "name" : "Abnormal Value",
>   "comment" : "FORMAT('For %s; the value %s exceeds threshold of %d',
> hostname, value, value_threshold)"
>   "rule" : "value > value_threshold",
>   "score" : 10
> }
>   ],
>   "aggregator" : "MAX"
> }
>
> The reason:
>
>- It's integrated and stellar is our default scripting layer
>- It supports doing some computation in the message
>
>
> On Wed, Feb 1, 2017 at 2:21 PM, Nick Allen  wrote:
>
> > Like I said, here is a proposed solution to one of the gaps I identified
> in
> > the previous email.
> >
> > *Problem*
> >
> > There is little transparency into the Threat Triage process itself.  When
> > Threat Triage runs, all I get is a score.  I don't know how that score
> was
> > arrived at, which rules were triggered, and the specific values that
> caused
> > a rule to trigger.
> >
> > More specifically, there is no way to generate a message that looks like
> > "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
> > threshold of '202'".  This makes it difficult for an analyst to action
> the
> > alert.
> >
> > *Proposed Solution*
> >
> > To improve the transparency of the Threat Triage process, I am proposing
> > these enhancements.
> >
> > 1. Threat Triage should attach to each message all of the rules that
> fired
> > in addition to the total calculated threat triage score.
> >
> > 2. Threat Triage should allow a custom message to be generated for each
> > rule.  The custom message would allow for some form of string
> interpolation
> > so that I can add specific values from each message to the generated
> > alert.  We could allow this in one or both of the new fields that Casey
> > just added, name and comment.
> >
> >
> > *Example*
> >
> > 1. In this example, we have a telemetry message with a field called
> 'value'
> > that we need to monitor.  In Enrichment, I calculate some sort of value
> > threshold, over which an alert should be generated.
> >
> >
> > 2. In Threat Triage, I use the calculated value threshold to alert on any
> > message that has a value exceeding this threshold.
> >
> > 3. I can embed values from the message, like the hostname, value, and
> value
> > threshold, into the alert produced by Threat Triage.  Notice that I am
> > using ${this} for string interpolation, but it could be any syntax that
> we
> > choose.
> >
> >
> > "triageConfig" : {
> >   "riskLevelRules" : [
> > {
> >   "name" : "Abnormal Value",
> >   "comment" : "For ${hostname}; the value ${value} exceeds threshold
> of
> > ${value_threshold}",
> >   "rule" : "value > value_threshold",
> >   "score" : 10
> > }
> >   ],
> >   "aggregator" : "MAX"
> > }
> >
> >
> > 4. The Threat Triage process today would add only the total calculated
> > score.
> >
> > "threat.triage.level": 10.0
> >
> >
> > With this proposal, Threat Triage would add the following to the message.
> >
> > Notice how each of the ${variables} have been replaced with the actual
> > values extracted from the message.  This allows for more contextual
> > information to action the alert.
> >
> > "threat.triage": {
> > "score": 10.0,
> > "rules": [
> >   {
> > "name": "Abnormal Value",
> > "comment" : "For 10.0.0.1; the value 101 exceeds threshold of
> 42",
> > "score" : 10
> >   }
> > ]
> > }
> >
> >
> >
> > What do you think?  Any alternative ideas?
> >
> >
> >
> > On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:
> >
> > > I'd like to explore the functionality that we have in Metron using a
> > > motivating example.  I think this will help highlight some gaps where
> we
> > > can enhance Metron.
> > >
> > > The motivating example is that I would like to create an alert if the
> > > number of inbound flows to any host over a 15 minute interval is
> > abnormal.
> > > I would like the alert to contain the specific information below to
> > > streamline the triage process.
> > >
> > > Rule: Abnormal number of inbound flows
> > > Bin: 15 mins
> > > Alert: The host 'powned.svr.bank.com' has '230' inbound flows,
> exceeding
> > > the threshold of '202'
> > >
> > >
> > > *What Works*
> > >
> > > In some ways, this example is similar to the "Outlier Detection" demo
> > that
> > > I performed with the Profiler a few months back.   We have most of what
> > we
> > > need to do this with a couple caveats.
> > >
> > > 1. An enrichment would be added to enrich the message with the corr

Re: [Discuss] Improve Alerting

[Nick Allen]

Great.  I think we're thinking along the same lines.  I just sent a
follow-up of another proposal that takes this idea a little further.  What
if we treated the Profiler as another source of telemetry?

On Wed, Feb 1, 2017 at 2:23 PM, Casey Stella  wrote:

> Regarding point 2, could we enable the profiler to write data to kafka and
> the enrichment queue?
>
> I'm proposing the profiler do something like this:
>
>- Count the number of inbound flows
>- On the tick, send a message to the enrichment queue containing:
>   - the number of flows
>   - A source type of 'system_alert'
>   - is_alert set to true
>- In enrichment, we enrich and triage system_alert source data in the
>same way we do any other.
>
> This would not solve the transparency issue, but at least make it so we
> keep triage in one place in the architecture.  Also, enabling kafka writing
> would enable other types of use-cases, like situations where we find
> outliers *directly* in the profile and send the alerts directly to the
> indexing queue without triage.
>
> The only changes this proposal would require would be
>
>1. a "write" section to a profile that takes a list of stellar
>statements and gets run on the tick write
>2. fixing the kafka writing stellar functions
>
> Casey
>
> On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:
>
> > I'd like to explore the functionality that we have in Metron using a
> > motivating example.  I think this will help highlight some gaps where we
> > can enhance Metron.
> >
> > The motivating example is that I would like to create an alert if the
> > number of inbound flows to any host over a 15 minute interval is
> abnormal.
> > I would like the alert to contain the specific information below to
> > streamline the triage process.
> >
> > Rule: Abnormal number of inbound flows
> > Bin: 15 mins
> > Alert: The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
> > the threshold of '202'
> >
> >
> > *What Works*
> >
> > In some ways, this example is similar to the "Outlier Detection" demo
> that
> > I performed with the Profiler a few months back.   We have most of what
> we
> > need to do this with a couple caveats.
> >
> > 1. An enrichment would be added to enrich the message with the correct
> > internal hostname 'powned.svr.bank.com'.
> >
> > 2. With the Profiler, I can capture some idea of what "normal" is for the
> > number of inbound flows across 15 minute intervals.
> > 3. With Threat Triage, I can create rules that alert when a value exceeds
> > what the Profiler defines as normal.
> >
> >
> > *What's Missing*
> >
> > Its nice to know that we are almost all the way there with this example.
> > Unfortunately, there are two gaps that fall out of this.
> >
> >  1. *Threat Triage Transparency*
> >
> > There is little transparency into the Threat Triage process itself.  When
> > Threat Triage runs, all I get is a score.  I don't know how that score
> was
> > arrived at, which rules were triggered, and the specific values that
> caused
> > a rule to trigger.
> >
> > More specifically, there is no way to generate a message that looks like
> > "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
> > threshold of '202'".
> >
> >
> > 2. *Triage Calculated Values from the Profiler*
> >
> > Also, the value being interrogated here, the number of inbound flows, is
> > not a static value contained within any single telemetry message.  This
> > value is calculated across multiple messages by the Profiler.  The
> current
> > Threat Triage process cannot be used to interrogate values calculated by
> > the Profiler.
> >
> >
> > To try and keep this email concise and digestible, I am going to send a
> > follow-on discussing proposed solutions for each of these separately.
> >
>



-- 
Nick Allen 

Re: [Discuss] Improve Alerting

[Nick Allen]

*Problem*

Triage Calculated Values from the Profiler

The value being interrogated here, the number of inbound flows, is not a
static value contained within any single telemetry message.  This value is
calculated across multiple messages by the Profiler.  The current Threat
Triage process cannot be used to interrogate values calculated by the
Profiler.

*Proposed Solution*

What I am proposing here is that we treat the Profiler as a source of
telemetry.   The measurements captured by the Profiler would be enqueued
into a Kafka topic.  We would then treat those Profiler messages like any
other telemetry.  We would parse, enrich, triage, and index those messages.

While this isn't fully formed in my head, I am throwing this out there as I
think its rather interesting.  I think this would have the following
advantages.

1.  We would be able to resuse the same threat triage mechanism for values
calculated by the Profiler.

2.  We would be able to generate profiles from the profiled data - aka
meta-profiles anyone?  I am really curious if this helps the MAD use case
at all.

3.  We could also potentially have all the data produced by the Profiler
written to HBase using the same indexing mechanism as all the other
telemetry.

On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:

> I'd like to explore the functionality that we have in Metron using a
> motivating example.  I think this will help highlight some gaps where we
> can enhance Metron.
>
> The motivating example is that I would like to create an alert if the
> number of inbound flows to any host over a 15 minute interval is abnormal.
> I would like the alert to contain the specific information below to
> streamline the triage process.
>
> Rule: Abnormal number of inbound flows
> Bin: 15 mins
> Alert: The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
> the threshold of '202'
>
>
> *What Works*
>
> In some ways, this example is similar to the "Outlier Detection" demo that
> I performed with the Profiler a few months back.   We have most of what we
> need to do this with a couple caveats.
>
> 1. An enrichment would be added to enrich the message with the correct
> internal hostname 'powned.svr.bank.com'.
>
> 2. With the Profiler, I can capture some idea of what "normal" is for the
> number of inbound flows across 15 minute intervals.
> 3. With Threat Triage, I can create rules that alert when a value exceeds
> what the Profiler defines as normal.
>
>
> *What's Missing*
>
> Its nice to know that we are almost all the way there with this example.
> Unfortunately, there are two gaps that fall out of this.
>
>  1. *Threat Triage Transparency*
>
> There is little transparency into the Threat Triage process itself.  When
> Threat Triage runs, all I get is a score.  I don't know how that score was
> arrived at, which rules were triggered, and the specific values that caused
> a rule to trigger.
>
> More specifically, there is no way to generate a message that looks like
> "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
> threshold of '202'".
>
>
> 2. *Triage Calculated Values from the Profiler*
>
> Also, the value being interrogated here, the number of inbound flows, is
> not a static value contained within any single telemetry message.  This
> value is calculated across multiple messages by the Profiler.  The current
> Threat Triage process cannot be used to interrogate values calculated by
> the Profiler.
>
>
> To try and keep this email concise and digestible, I am going to send a
> follow-on discussing proposed solutions for each of these separately.
>
>
>
>
>
>


-- 
Nick Allen 

Re: [Discuss] Improve Alerting

[Casey Stella]

I like the direction.  One thing that we may want is for comment to just be
a stellar expression and construct a function to essentially do
String.format().  So, that'd become:
"triageConfig" : {
  "riskLevelRules" : [
{
  "name" : "Abnormal Value",
  "comment" : "FORMAT('For %s; the value %s exceeds threshold of %d',
hostname, value, value_threshold)"
  "rule" : "value > value_threshold",
  "score" : 10
}
  ],
  "aggregator" : "MAX"
}

The reason:

   - It's integrated and stellar is our default scripting layer
   - It supports doing some computation in the message


On Wed, Feb 1, 2017 at 2:21 PM, Nick Allen  wrote:

> Like I said, here is a proposed solution to one of the gaps I identified in
> the previous email.
>
> *Problem*
>
> There is little transparency into the Threat Triage process itself.  When
> Threat Triage runs, all I get is a score.  I don't know how that score was
> arrived at, which rules were triggered, and the specific values that caused
> a rule to trigger.
>
> More specifically, there is no way to generate a message that looks like
> "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
> threshold of '202'".  This makes it difficult for an analyst to action the
> alert.
>
> *Proposed Solution*
>
> To improve the transparency of the Threat Triage process, I am proposing
> these enhancements.
>
> 1. Threat Triage should attach to each message all of the rules that fired
> in addition to the total calculated threat triage score.
>
> 2. Threat Triage should allow a custom message to be generated for each
> rule.  The custom message would allow for some form of string interpolation
> so that I can add specific values from each message to the generated
> alert.  We could allow this in one or both of the new fields that Casey
> just added, name and comment.
>
>
> *Example*
>
> 1. In this example, we have a telemetry message with a field called 'value'
> that we need to monitor.  In Enrichment, I calculate some sort of value
> threshold, over which an alert should be generated.
>
>
> 2. In Threat Triage, I use the calculated value threshold to alert on any
> message that has a value exceeding this threshold.
>
> 3. I can embed values from the message, like the hostname, value, and value
> threshold, into the alert produced by Threat Triage.  Notice that I am
> using ${this} for string interpolation, but it could be any syntax that we
> choose.
>
>
> "triageConfig" : {
>   "riskLevelRules" : [
> {
>   "name" : "Abnormal Value",
>   "comment" : "For ${hostname}; the value ${value} exceeds threshold of
> ${value_threshold}",
>   "rule" : "value > value_threshold",
>   "score" : 10
> }
>   ],
>   "aggregator" : "MAX"
> }
>
>
> 4. The Threat Triage process today would add only the total calculated
> score.
>
> "threat.triage.level": 10.0
>
>
> With this proposal, Threat Triage would add the following to the message.
>
> Notice how each of the ${variables} have been replaced with the actual
> values extracted from the message.  This allows for more contextual
> information to action the alert.
>
> "threat.triage": {
> "score": 10.0,
> "rules": [
>   {
> "name": "Abnormal Value",
> "comment" : "For 10.0.0.1; the value 101 exceeds threshold of 42",
> "score" : 10
>   }
> ]
> }
>
>
>
> What do you think?  Any alternative ideas?
>
>
>
> On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:
>
> > I'd like to explore the functionality that we have in Metron using a
> > motivating example.  I think this will help highlight some gaps where we
> > can enhance Metron.
> >
> > The motivating example is that I would like to create an alert if the
> > number of inbound flows to any host over a 15 minute interval is
> abnormal.
> > I would like the alert to contain the specific information below to
> > streamline the triage process.
> >
> > Rule: Abnormal number of inbound flows
> > Bin: 15 mins
> > Alert: The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
> > the threshold of '202'
> >
> >
> > *What Works*
> >
> > In some ways, this example is similar to the "Outlier Detection" demo
> that
> > I performed with the Profiler a few months back.   We have most of what
> we
> > need to do this with a couple caveats.
> >
> > 1. An enrichment would be added to enrich the message with the correct
> > internal hostname 'powned.svr.bank.com'.
> >
> > 2. With the Profiler, I can capture some idea of what "normal" is for the
> > number of inbound flows across 15 minute intervals.
> > 3. With Threat Triage, I can create rules that alert when a value exceeds
> > what the Profiler defines as normal.
> >
> >
> > *What's Missing*
> >
> > Its nice to know that we are almost all the way there with this example.
> > Unfortunately, there are two gaps that fall out of this.
> >
> >  1. *Threat Triage Transparency*
> >
> > There is little transparency into the Threat Triage process itself.  When
> > Threat Triag

Re: [Discuss] Improve Alerting

[Casey Stella]

Regarding point 2, could we enable the profiler to write data to kafka and
the enrichment queue?

I'm proposing the profiler do something like this:

   - Count the number of inbound flows
   - On the tick, send a message to the enrichment queue containing:
  - the number of flows
  - A source type of 'system_alert'
  - is_alert set to true
   - In enrichment, we enrich and triage system_alert source data in the
   same way we do any other.

This would not solve the transparency issue, but at least make it so we
keep triage in one place in the architecture.  Also, enabling kafka writing
would enable other types of use-cases, like situations where we find
outliers *directly* in the profile and send the alerts directly to the
indexing queue without triage.

The only changes this proposal would require would be

   1. a "write" section to a profile that takes a list of stellar
   statements and gets run on the tick write
   2. fixing the kafka writing stellar functions

Casey

On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:

> I'd like to explore the functionality that we have in Metron using a
> motivating example.  I think this will help highlight some gaps where we
> can enhance Metron.
>
> The motivating example is that I would like to create an alert if the
> number of inbound flows to any host over a 15 minute interval is abnormal.
> I would like the alert to contain the specific information below to
> streamline the triage process.
>
> Rule: Abnormal number of inbound flows
> Bin: 15 mins
> Alert: The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
> the threshold of '202'
>
>
> *What Works*
>
> In some ways, this example is similar to the "Outlier Detection" demo that
> I performed with the Profiler a few months back.   We have most of what we
> need to do this with a couple caveats.
>
> 1. An enrichment would be added to enrich the message with the correct
> internal hostname 'powned.svr.bank.com'.
>
> 2. With the Profiler, I can capture some idea of what "normal" is for the
> number of inbound flows across 15 minute intervals.
> 3. With Threat Triage, I can create rules that alert when a value exceeds
> what the Profiler defines as normal.
>
>
> *What's Missing*
>
> Its nice to know that we are almost all the way there with this example.
> Unfortunately, there are two gaps that fall out of this.
>
>  1. *Threat Triage Transparency*
>
> There is little transparency into the Threat Triage process itself.  When
> Threat Triage runs, all I get is a score.  I don't know how that score was
> arrived at, which rules were triggered, and the specific values that caused
> a rule to trigger.
>
> More specifically, there is no way to generate a message that looks like
> "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
> threshold of '202'".
>
>
> 2. *Triage Calculated Values from the Profiler*
>
> Also, the value being interrogated here, the number of inbound flows, is
> not a static value contained within any single telemetry message.  This
> value is calculated across multiple messages by the Profiler.  The current
> Threat Triage process cannot be used to interrogate values calculated by
> the Profiler.
>
>
> To try and keep this email concise and digestible, I am going to send a
> follow-on discussing proposed solutions for each of these separately.
>

Re: [Discuss] Improve Alerting

[Nick Allen]

Like I said, here is a proposed solution to one of the gaps I identified in
the previous email.

*Problem*

There is little transparency into the Threat Triage process itself.  When
Threat Triage runs, all I get is a score.  I don't know how that score was
arrived at, which rules were triggered, and the specific values that caused
a rule to trigger.

More specifically, there is no way to generate a message that looks like
"The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
threshold of '202'".  This makes it difficult for an analyst to action the
alert.

*Proposed Solution*

To improve the transparency of the Threat Triage process, I am proposing
these enhancements.

1. Threat Triage should attach to each message all of the rules that fired
in addition to the total calculated threat triage score.

2. Threat Triage should allow a custom message to be generated for each
rule.  The custom message would allow for some form of string interpolation
so that I can add specific values from each message to the generated
alert.  We could allow this in one or both of the new fields that Casey
just added, name and comment.


*Example*

1. In this example, we have a telemetry message with a field called 'value'
that we need to monitor.  In Enrichment, I calculate some sort of value
threshold, over which an alert should be generated.


2. In Threat Triage, I use the calculated value threshold to alert on any
message that has a value exceeding this threshold.

3. I can embed values from the message, like the hostname, value, and value
threshold, into the alert produced by Threat Triage.  Notice that I am
using ${this} for string interpolation, but it could be any syntax that we
choose.


"triageConfig" : {
  "riskLevelRules" : [
{
  "name" : "Abnormal Value",
  "comment" : "For ${hostname}; the value ${value} exceeds threshold of
${value_threshold}",
  "rule" : "value > value_threshold",
  "score" : 10
}
  ],
  "aggregator" : "MAX"
}


4. The Threat Triage process today would add only the total calculated
score.

"threat.triage.level": 10.0


With this proposal, Threat Triage would add the following to the message.

Notice how each of the ${variables} have been replaced with the actual
values extracted from the message.  This allows for more contextual
information to action the alert.

"threat.triage": {
"score": 10.0,
"rules": [
  {
"name": "Abnormal Value",
"comment" : "For 10.0.0.1; the value 101 exceeds threshold of 42",
"score" : 10
  }
]
}



What do you think?  Any alternative ideas?



On Wed, Feb 1, 2017 at 2:11 PM, Nick Allen  wrote:

> I'd like to explore the functionality that we have in Metron using a
> motivating example.  I think this will help highlight some gaps where we
> can enhance Metron.
>
> The motivating example is that I would like to create an alert if the
> number of inbound flows to any host over a 15 minute interval is abnormal.
> I would like the alert to contain the specific information below to
> streamline the triage process.
>
> Rule: Abnormal number of inbound flows
> Bin: 15 mins
> Alert: The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
> the threshold of '202'
>
>
> *What Works*
>
> In some ways, this example is similar to the "Outlier Detection" demo that
> I performed with the Profiler a few months back.   We have most of what we
> need to do this with a couple caveats.
>
> 1. An enrichment would be added to enrich the message with the correct
> internal hostname 'powned.svr.bank.com'.
>
> 2. With the Profiler, I can capture some idea of what "normal" is for the
> number of inbound flows across 15 minute intervals.
> 3. With Threat Triage, I can create rules that alert when a value exceeds
> what the Profiler defines as normal.
>
>
> *What's Missing*
>
> Its nice to know that we are almost all the way there with this example.
> Unfortunately, there are two gaps that fall out of this.
>
>  1. *Threat Triage Transparency*
>
> There is little transparency into the Threat Triage process itself.  When
> Threat Triage runs, all I get is a score.  I don't know how that score was
> arrived at, which rules were triggered, and the specific values that caused
> a rule to trigger.
>
> More specifically, there is no way to generate a message that looks like
> "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
> threshold of '202'".
>
>
> 2. *Triage Calculated Values from the Profiler*
>
> Also, the value being interrogated here, the number of inbound flows, is
> not a static value contained within any single telemetry message.  This
> value is calculated across multiple messages by the Profiler.  The current
> Threat Triage process cannot be used to interrogate values calculated by
> the Profiler.
>
>
> To try and keep this email concise and digestible, I am going to send a
> follow-on discussing proposed solutions for each of these separately.
>
>
>
>
>
>


-- 
Nick Allen 

Re: [DISCUSS] Metron Management UI

[Houshang Livian]

Hello Dima,

Thank you for taking a few minutes to look at this new addition. I hope that it 
can make working with Metron much easier.

Do you think ElasticSearch Index Templates are a MUST have initially? Or is it 
okay to start here?

I do like the idea of adding that functionality.

Houshang




On 2/1/17, 6:51 AM, "Dima Kovalyov"  wrote:

>Thank you Houshang,
>
>That looks absolutely great, loved it!
>Will it make sense to also add ElasticSearch index  templates
>querying/creation here?
>
>- Dima 
>
>On 02/01/2017 02:06 AM, Houshang Livian wrote:
>> Hello Metron Community,
>>
>> We have constructed a Management Module UI, built on top of METRON-503 (REST 
>> API) (currently under review). This Module gives users the ability to setup 
>> and administer much of the product through the UI.
>>
>> Here are some screens to show you what we are thinking. Please take a look:
>> http://imgur.com/a/QAQyu?
>>
>> Does this look like a reasonable place to start?
>> Is there anything that is an absolute MUST have or MUST NOT have?
>>
>> Houshang Livian
>> Senior User Experience Designer
>> Hortonworks
>>
>> www.hortonworks.com
>> 
>>
>> Mobile: (831) 521-4176
>> hliv...@hortonworks.com
>> 
>>
>>
>
>

[Discuss] Improve Alerting

[Nick Allen]

I'd like to explore the functionality that we have in Metron using a
motivating example.  I think this will help highlight some gaps where we
can enhance Metron.

The motivating example is that I would like to create an alert if the
number of inbound flows to any host over a 15 minute interval is abnormal.
I would like the alert to contain the specific information below to
streamline the triage process.

Rule: Abnormal number of inbound flows
Bin: 15 mins
Alert: The host 'powned.svr.bank.com' has '230' inbound flows, exceeding
the threshold of '202'


*What Works*

In some ways, this example is similar to the "Outlier Detection" demo that
I performed with the Profiler a few months back.   We have most of what we
need to do this with a couple caveats.

1. An enrichment would be added to enrich the message with the correct
internal hostname 'powned.svr.bank.com'.

2. With the Profiler, I can capture some idea of what "normal" is for the
number of inbound flows across 15 minute intervals.
3. With Threat Triage, I can create rules that alert when a value exceeds
what the Profiler defines as normal.


*What's Missing*

Its nice to know that we are almost all the way there with this example.
Unfortunately, there are two gaps that fall out of this.

 1. *Threat Triage Transparency*

There is little transparency into the Threat Triage process itself.  When
Threat Triage runs, all I get is a score.  I don't know how that score was
arrived at, which rules were triggered, and the specific values that caused
a rule to trigger.

More specifically, there is no way to generate a message that looks like
"The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the
threshold of '202'".


2. *Triage Calculated Values from the Profiler*

Also, the value being interrogated here, the number of inbound flows, is
not a static value contained within any single telemetry message.  This
value is calculated across multiple messages by the Profiler.  The current
Threat Triage process cannot be used to interrogate values calculated by
the Profiler.


To try and keep this email concise and digestible, I am going to send a
follow-on discussing proposed solutions for each of these separately.

[GitHub] incubator-metron issue #430: METRON-658: Updated Grammar to Handle More Uses...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/430
  
+1 by inspection, good contribution.  I like the refactoring of the grammar 
too.  Stellar is getting cleaner and cleaner with every contribution like this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #426: METRON-675: Make Threat Triage rules abl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/426


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #426: METRON-675: Make Threat Triage rules able to be...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/426
  
@nickwallen Agreed on the aggregator being stellar.  I created 
https://issues.apache.org/jira/browse/METRON-683 for it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/432
  
# Testing Plan

## Preliminaries

* Download the alexa 1m dataset:
```
wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
unzip top-1m.csv.zip
```
* Stage import files
```
head -n 1 top-1m.csv > top-10k.csv
hadoop fs -put top-10k.csv /tmp
head -n 1 top-1m.csv | gzip - > top-10k.csv.gz
head -n 1 top-1m.csv | zip > top-10k.csv.zip
```
* Create an extractor.json for the CSV data by editing `extractor.json` and 
pasting in these contents:
```
{
  "config" : {
"columns" : {
   "domain" : 1,
   "rank" : 0
}
,"indicator_column" : "domain"
,"type" : "alexa"
,"separator" : ","
 },
  "extractor" : "CSV"
}
```

## Import from URL
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase from URL.  This should take approximately 5 or 6 
minutes
/usr/metron/0.3.0/bin/flatfile_loader.sh -i 
http://s3.amazonaws.com/alexa-static/top-1m.csv.zip -t enrichment -c t -e 
./extractor.json -p 5 -b 128
# count data written and verify it's 1M
echo "count 'enrichment'" | hbase shell
```

## Import from local file (non-zipped)
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase 
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-10k.csv -t enrichment -c 
t -e ./extractor.json -p 5 -b 128
# count data written and verify it's 10k
echo "count 'enrichment'" | hbase shell
```

## Import from local file (gzipped)
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase 
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-10k.csv.gz -t enrichment 
-c t -e ./extractor.json -p 5 -b 128
# count data written and verify it's 10k
echo "count 'enrichment'" | hbase shell
```

## Import from local file (zipped)
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase 
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-10k.csv.zip -t enrichment 
-c t -e ./extractor.json -p 5 -b 128
# count data written and verify it's 10k
echo "count 'enrichment'" | hbase shell
```

## Import from HDFS via MR
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase 
/usr/metron/0.3.0/bin/flatfile_loader.sh -i /tmp/top-10k.csv -t enrichment 
-c t -e ./extractor.json -m MR
# count data written and verify it's 10k
echo "count 'enrichment'" | hbase shell
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/432
  
I know it seems like a lot of code changed, but a lot of this was 
reorganizing the flat file loader class and splitting it into separate reusable 
components, rather than new code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/432

METRON-682: Unify and Improve the Flat File Loader

Currently the flat file loader is deficient in a couple ways:
* It only supports importing local data despite there being a separate, 
poorly named, application which supports importing enrichment via MapReduce 
called threat_intel_loader.sh
* It does not support local imports from HDFS
* It does not support local imports from URLs
* It does not support importing zipped archives locally
* You cannot import more than one file at once

This JIRA will:
* Unify the MapReduce and local imports into one program and allow the user 
to specify the import mode with a CLI flag
* Support local imports from HDFS and URLs
* Support local imports from zipped files
* Support importing more than one file at once

Testing plan in comments

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron unified_loader

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/432.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #432






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Metron Management UI

[Kyle Richardson]

This looks awesome! A great starting point. Like Jon, I'm looking forward
to kicking the tires.

-Kyle

On Wed, Feb 1, 2017 at 10:18 AM, Ryan Merriman  wrote:

> Jon, I've done my best to keep this UI in sync with the API PR so you can
> play around with it now if you want.  There are 2 different versions of the
> UI:  one without the Blockly editor (
> https://github.com/merrimanr/incubator-metron/tree/METRON-623) and one
> with
> (https://github.com/merrimanr/incubator-metron/tree/blockly).  The Blockly
> editor feature contains a significant amount of code so we decide to keep
> it in it's own branch so reviewing will be easier.
>
> If you are familiar with the javascript ecosystem it's not hard to start it
> up but some instructions will definitely help.  I will add a README to make
> this easier but here's a quick summary:
>
> Start up the Docker environment in metron-docker
> Set the "docker.host.address" property in
> /incubator-metron/metron-interface/metron-rest/src/
> main/resources/application-docker.yml
> to match the IP address of your Docker machine
> Run org.apache.metron.rest.MetronRestApplication with spring active
> profiles set to "docker,dev"
> Navigate to metron-interface/metron-config and run "npm install"
> Start the UI with "metron-config/scripts/start-dev.sh"
>
> We're still working on a seamless install process but this should work in
> the meantime.  Feel free to reach out if you need help getting it going.
> As for feedback, I wouldn't limit it to what the API currently provides the
> API PR.  We can add services as needed.
>
> Ryan
>
> On Tue, Jan 31, 2017 at 7:49 PM, zeo...@gmail.com 
> wrote:
>
> > First off - this is an awesome first take at a management UI and I'm
> > looking forward to messing around with it.  Other than skimming some of
> the
> > dialogue as it comes in I have not been keeping up with the API PR.
> Should
> > I be able to assume that the UI PR is broken until the API is merged and
> > the UI can be updated for any changes?  I have not spun up the UI based
> on
> > that assumption.
> >
> > Are you looking for feedback that's limited to what can currently be done
> > using the API (a CRUD interface for SensorParserConfigs)?
> >
> > Thanks,
> >
> > Jon
> >
> > On Tue, Jan 31, 2017, 7:06 PM Houshang Livian 
> > wrote:
> >
> > Hello Metron Community,
> >
> > We have constructed a Management Module UI, built on top of METRON-503
> > (REST API) (currently under review). This Module gives users the ability
> to
> > setup and administer much of the product through the UI.
> >
> > Here are some screens to show you what we are thinking. Please take a
> look:
> > http://imgur.com/a/QAQyu?
> >
> > Does this look like a reasonable place to start?
> > Is there anything that is an absolute MUST have or MUST NOT have?
> >
> > Houshang Livian
> > Senior User Experience Designer
> > Hortonworks
> >
> > www.hortonworks.com
> > 
> >
> > Mobile: (831) 521-4176
> > hliv...@hortonworks.com
> > 
> >
> > --
> >
> > Jon
> >
> > Sent from my mobile device
> >
>

Re: [DISCUSS] Metron Management UI

[Ryan Merriman]

Jon, I've done my best to keep this UI in sync with the API PR so you can
play around with it now if you want.  There are 2 different versions of the
UI:  one without the Blockly editor (
https://github.com/merrimanr/incubator-metron/tree/METRON-623) and one with
(https://github.com/merrimanr/incubator-metron/tree/blockly).  The Blockly
editor feature contains a significant amount of code so we decide to keep
it in it's own branch so reviewing will be easier.

If you are familiar with the javascript ecosystem it's not hard to start it
up but some instructions will definitely help.  I will add a README to make
this easier but here's a quick summary:

Start up the Docker environment in metron-docker
Set the "docker.host.address" property in
/incubator-metron/metron-interface/metron-rest/src/main/resources/application-docker.yml
to match the IP address of your Docker machine
Run org.apache.metron.rest.MetronRestApplication with spring active
profiles set to "docker,dev"
Navigate to metron-interface/metron-config and run "npm install"
Start the UI with "metron-config/scripts/start-dev.sh"

We're still working on a seamless install process but this should work in
the meantime.  Feel free to reach out if you need help getting it going.
As for feedback, I wouldn't limit it to what the API currently provides the
API PR.  We can add services as needed.

Ryan

On Tue, Jan 31, 2017 at 7:49 PM, zeo...@gmail.com  wrote:

> First off - this is an awesome first take at a management UI and I'm
> looking forward to messing around with it.  Other than skimming some of the
> dialogue as it comes in I have not been keeping up with the API PR.  Should
> I be able to assume that the UI PR is broken until the API is merged and
> the UI can be updated for any changes?  I have not spun up the UI based on
> that assumption.
>
> Are you looking for feedback that's limited to what can currently be done
> using the API (a CRUD interface for SensorParserConfigs)?
>
> Thanks,
>
> Jon
>
> On Tue, Jan 31, 2017, 7:06 PM Houshang Livian 
> wrote:
>
> Hello Metron Community,
>
> We have constructed a Management Module UI, built on top of METRON-503
> (REST API) (currently under review). This Module gives users the ability to
> setup and administer much of the product through the UI.
>
> Here are some screens to show you what we are thinking. Please take a look:
> http://imgur.com/a/QAQyu?
>
> Does this look like a reasonable place to start?
> Is there anything that is an absolute MUST have or MUST NOT have?
>
> Houshang Livian
> Senior User Experience Designer
> Hortonworks
>
> www.hortonworks.com
> 
>
> Mobile: (831) 521-4176
> hliv...@hortonworks.com
> 
>
> --
>
> Jon
>
> Sent from my mobile device
>

[DISCUSS] Metron GUI Syleguide

[Raghu Mitra Kandikonda]

I would like to start a discussion around what would be a good style guide for 
a angular/SASS/bootstrap based GUI for metron.

A coding styleguide (note, not a visual styleguide) is a valuable tool for 
teams that strives for a degree of standardization in their code. A good 
styleguide, when well followed, will

-set the standard for code quality across a codebase;
-promote consistency across codebases;
-give developers a feeling of familiarity across codebases;
-increase productivity.

ANGULAR:
A lot of guidelines are already proposed by angular team in their docs which 
when followed creates a good consistent code base. You can check the angular2 
style guide here. 
While the guide talks in length about anguar code(Javascript/Typescript)  it 
doesn't talk about standards for html and css and hence creating the need of 
this doc.

HTML & CSS:
There are many good style guides out there we can follow them as is or cherry 
pick the items as we like. I would propose that we follow these below two 
guides as is.

https://google.github.io/styleguide/htmlcssguide.xml
https://github.com/airbnb/css

You can look at more style guides at http://styleguides.io/

While these style guides talk about general HTML/CSS we can have few more 
guidelines around angular component templates and styles.
1.We can follow BEM naming conventions for all css files 
(https://css-tricks.com/bem-101/)
2. Keep the style of the component within the component, do not add it to the 
global styles
3. Prefer overwriting the styles instead of creating a new class.
Ex: if we have a global style as .btn{min-width:100px} and we want a button to 
have min-width of 200px in the component style you can overwrite it as 
.btn{min-width:200px} instead of defining new class like 
.btn-someother{min-width:200px} this will ensure that we can have a common 
place to add any new attributes to buttons in the future.
4.Add all the colors and any other variables that are used in theming of the 
appication to _variables.scss. This will help us to re-theme the application.
5.Use the bootstrap components and layouts as much as possible.


-Regards,
Raghu Mitra

Re: [DISCUSS] Metron Management UI

[Dima Kovalyov]

Thank you Houshang,

That looks absolutely great, loved it!
Will it make sense to also add ElasticSearch index  templates
querying/creation here?

- Dima 

On 02/01/2017 02:06 AM, Houshang Livian wrote:
> Hello Metron Community,
>
> We have constructed a Management Module UI, built on top of METRON-503 (REST 
> API) (currently under review). This Module gives users the ability to setup 
> and administer much of the product through the UI.
>
> Here are some screens to show you what we are thinking. Please take a look:
> http://imgur.com/a/QAQyu?
>
> Does this look like a reasonable place to start?
> Is there anything that is an absolute MUST have or MUST NOT have?
>
> Houshang Livian
> Senior User Experience Designer
> Hortonworks
>
> www.hortonworks.com
> 
>
> Mobile: (831) 521-4176
> hliv...@hortonworks.com
> 
>
>

[GitHub] incubator-metron pull request #431: METRON-680: GeoLiteDatabase incorrectly ...

[justinleet]

Github user justinleet closed the pull request at:

https://github.com/apache/incubator-metron/pull/431


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Error Indexing

[Dima Kovalyov]

That's a great topic of discussion.

Throughout the thread the idea of having hash of the message that failed
is changed, can someone please explain why do you plan to use this hash
and how?

- Dima

On 02/01/2017 06:23 AM, zeo...@gmail.com wrote:
> After thinking on this for a few days I recant my previous suggestion of
> TupleHash256.  It's still a bit early for SHA-3 - no good reference
> implementations/libraries exist (I did some searching and emailing), it is
> optimized for hardware but no hardware implementation is widely accessible,
> FIPS 140-3 is still not close to finalized, etc.
>
> I think we could simulate the benefits of tuplehash by sorting the tuples,
> then doing SHA-256(len(tuple1) | tuple1 | ... | len(tuplen) | tuplen).
> Happy to entertain opposing thoughts, such as BLAKE2, etc. but with the
> likely users of Metron, I think sticking with FIPS 140-2 is a solid choice.
>
> Jon
>
> On Thu, Jan 26, 2017, 11:23 AM zeo...@gmail.com  wrote:
>
> So one more thing regarding why I think we should throw an exception on a
> failed enrichment.  If we do make something like username a constant field,
> in cases where that is used to calculate rawMessage_hash, if it fails to
> enrich, the hash would be different compared to when it succeeds.  Of
> course I think the initial intent of adding username as a constant field
> would be to handle it in the parsers, where that information is provided in
> the messages themselves, but how would Threat Intel know the difference?
> In my environment I am looking forward to a streaming enrichment that adds
> the username, where applicable, anywhere I have an IP.
>
> My hesitant suggestion for a hashing algorithm would be to use
> TupleHash256, as it is a NIST-provided implementation of SHA-3 (using
> cSHAKE) for this use case.  Details here
> .
> However, I haven't been able to find a reference implementation of this in
> any language, so that's a bit of a downside.  A more general SHA3-256
> implementation where we handle ordering could work as well, but would be
> significantly less optimal.
>
> Jon
>
> On Thu, Jan 26, 2017 at 10:20 AM Ryan Merriman  wrote:
>
> Jon, I misread the code in the GenericEnrichmentBolt.  The error is
> forwarded on so no issues there.
>
> Defaulting to the common fields makes sense.  I will dig into the
> GenericEnrichmentBolt more, maybe there is a way to get the error fields
> without having to significantly change things.  Any opinion on a hashing
> algorithm?
>
> On Wed, Jan 25, 2017 at 9:37 PM, zeo...@gmail.com  wrote:
>
>> Although hashing the whole message is better than nothing, it misses a lot
>> of the benefits we could get.
>>
>> While I'd love to have consistency for this field across all of the
>> different error.types, it appears that may not be reasonably possible
>> because of the parsers.  So, how about something like hash all of the
>> constant
>> fields
>> > metron-platform/metron-common/src/main/java/org/apache/
>> metron/common/Constants.java>
>> excluding
>> timestamp and original_string unless it is a parser, in which case hash
> the
>> entire message?  This gives us some measure of event uniqueness and it can
>> grow as we define additional constant fields (I recall discussing with
>> someone else on the list regarding expanding those standard fields to
>> include things like usernames but I can't find the specific email
>> exchange).
>>
>> Because some enrichments can be heavily relied on, I think it makes sense
>> to put a message onto the error queue when it throws an exception.  Not
>> only does this help troubleshoot edge cases, but it makes issues more
>> obvious when assembling a new enrichment in dev/test.  I can't think of a
>> scenario currently where an enrichment would only be "best effort" and
> that
>> I wouldn't want that error indexed and retrievable.  However, this gets
>> interesting when talking about the various options to solve the "Enrich
>> enrichment" discussion from earlier in the month.  We can keep that part
> of
>> this separate though, as I don't think that's being actively pursued right
>> now.
>>
>> Jon
>>
>> On Wed, Jan 25, 2017 at 10:49 AM David Lyle  wrote:
>>
>> RE: separate JIRA for MPack/Ansible. No objection to tracking them
>> separately, but for this item to be complete, you'll need both the feature
>> and the ability to install it.
>>
>> -D...
>>
>>
>> On Tue, Jan 24, 2017 at 5:33 PM, Ryan Merriman 
>> wrote:
>>
>>> Assuming we're going to write all errors to a single error topic, I
> think
>>> it makes sense to agree on an error message schema and handle errors
>> across
>>> the 3 different topologies in the same way with a single implementation.
>>> The implementation in ParserBolt (ErrorUtils.handleError) produces the
>> most
>>> verbose error object so I think it's a good candidate for the single
>>> implementation.  Here is the mess