[GitHub] incubator-metron pull request #:

[bmorphism]

Github user bmorphism commented on the pull request:


https://github.com/apache/incubator-metron/commit/d1fcda6043fe608167cde616214e4059663fb131#commitcomment-20585942
  
Wonderful! Thank you, this helps with testing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #420: METRON-668: Remove the "tickUpdate" prof...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/420


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #420: METRON-668: Remove the "tickUpdate" prof...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/420

METRON-668: Remove the "tickUpdate" profile config and make the "init" 
phase not reset variables

Please see description at 
[METRON-668](https://issues.apache.org/jira/browse/METRON-668) for a full 
description.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-668

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #420


commit 7cb5c60462448b6c35a9d1def58489903a649834
Author: cstella 
Date:   2017-01-19T21:58:27Z

METRON-668: Remove the 'tickUpdate' profile config and make the 'init' 
phase not reset variables

commit ad94ed4916605b1416c62463fc2de96f76c85f9f
Author: cstella 
Date:   2017-01-19T22:00:47Z

Fixed docs

commit a99c013a3e850a26720ad9a7bfede226660e18ed
Author: cstella 
Date:   2017-01-23T19:07:24Z

Updating unit tests to function properly.

commit 78a472fda1e65c0679089127d149582f861c82a7
Author: cstella 
Date:   2017-01-23T20:56:20Z

TEMPORARY UPDATE TO SEE SOMETHING, DO NOT MERGE YET

commit 2d4aa56324f86cc5e3b73d486a2fde008ec211b8
Author: cstella 
Date:   2017-01-23T21:24:32Z

updating one more time to indicate writes vs flush

commit b78e31fef6244aba98b369553125c8c667ff0a35
Author: cstella 
Date:   2017-01-23T21:27:02Z

updating again.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[DISCUSS] Gratuating to Apache Top Level Project

[James Sirota]

I think the Apache Incubation was very valuable learning experience for us, but 
it seems like we are ready to become a top-level project.  We have been in 
incubation since 2015-12-06 and under the guidance of our Mentors we had a 
clean build (0.3.0), we learned how to function well as a community (as defined 
by Apache), and if you look at our maturity level checklist we meet the 
criteria of the a mature Apache project. 
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66852119)


Do you think we are ready to graduate?  Should we start putting a case together 
for graduation?

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: [DISCUSS] Error Indexing

[zeo...@gmail.com]

In that case the hash would be of the value in the IP field, such as
sha3(8.8.8.8).

Jon

On Mon, Jan 23, 2017, 6:41 PM James Sirota  wrote:

> Jon,
>
> I am still not entirely following why we would want to use hashing.  For
> example if my error is "Your IP field is invalid and failed validation"
> hashing this error string will always result in the same hash.  Why not
> just use the actual error string? Can you provide an example where you
> would use it?
>
> Thanks,
> James
>
> 23.01.2017, 16:29, "zeo...@gmail.com" :
> > For 1 - I'm good with that.
> >
> > I'm talking about hashing the relevant content itself not the error. Some
> > benefits are (1) minimize load on search index (there's minimal benefit
> in
> > spending the CPU and disk to keep it at full fidelity (tokenize and
> store))
> > (2) provide something to key on for dashboards (assuming a good hash
> > algorithm that avoids collisions and is second preimage resistant) and
> (3)
> > specific to errors, if the issue is that it failed to index, a hash gives
> > us some protection that the issue will not occur twice.
> >
> > Jon
> >
> > On Mon, Jan 23, 2017, 2:47 PM James Sirota  wrote:
> >
> > Jon,
> >
> > With regards to 1, collapsing to a single dashboard for each would be
> > fine. So we would have one error index and one "failed to validate"
> > index. The distinction is that errors would be things that went wrong
> > during stream processing (failed to parse, etc...), while validation
> > failures are messages that explicitly failed stellar validation/schema
> > enforcement. There should be relatively few of the second type.
> >
> > With respect to 3, why do you want the error hashed? Why not just search
> > for the error text?
> >
> > Thanks,
> > James
> >
> > 20.01.2017, 14:01, "zeo...@gmail.com" :
> >>  As someone who currently fills the platform engineer role, I can give
> this
> >>  idea a huge +1. My thoughts:
> >>
> >>  1. I think it depends on exactly what data is pushed into the index
> (#3).
> >>  However, assuming the errors you proposed recording, I can't see huge
> >>  benefits to having more than one dashboard. I would be happy to be
> >>  persuaded otherwise.
> >>
> >>  2. I would say yes, storing the errors in HDFS in addition to indexing
> is
> >>  a good thing. Using METRON-510
> >>   as a case study,
> there
> >>  is the potential in this environment for attacker-controlled data to
> >
> > result
> >>  in processing errors which could be a method of evading security
> >>  monitoring. Once an attack is identified, the long term HDFS storage
> would
> >>  allow better historical analysis for low-and-slow/persistent attacks
> (I'm
> >>  thinking of a method of data exfil that also won't successfully get
> stored
> >>  in Lucene, but is hard to identify over a short period of time).
> >>   - Along this line, I think that there are various parts of Metron
> (this
> >>  included) which could benefit from having method of configuring data
> aging
> >>  by bucket in HDFS (Following Nick's comments here
> >>  ).
> >>
> >>  3. I would potentially add a hash of the content that failed
> validation to
> >>  help identify repeats over time with less of a concern that you'd have
> >
> > back
> >>  to back failures (i.e. instead of storing the value itself).
> Additionally,
> >>  I think it's helpful to be able to search all times there was an
> indexing
> >>  error (instead of it hitting the catch-all).
> >>
> >>  Jon
> >>
> >>  On Fri, Jan 20, 2017 at 1:17 PM James Sirota 
> wrote:
> >>
> >>  We already have a capability to capture bolt errors and validation
> errors
> >>  and pipe them into a Kafka topic. I want to propose that we attach a
> >>  writer topology to the error and validation failed kafka topics so
> that we
> >>  can (a) create a new ES index for these errors and (b) create a new
> Kibana
> >>  dashboard to visualize them. The benefit would be that errors and
> >>  validation failures would be easier to see and analyze.
> >>
> >>  I am seeking feedback on the following:
> >>
> >>  - How granular would we want this feature to be? Think we would want
> one
> >>  index/dashboard per source? Or would it be better to collapse
> everything
> >>  into the same index?
> >>  - Do we care about storing these errors in HDFS as well? Or is indexing
> >>  them enough?
> >>  - What types of errors should we record? I am proposing:
> >>
> >>  For error reporting:
> >>  --Message failed to parse
> >>  --Enrichment failed to enrich
> >>  --Threat intel feed failures
> >>  --Generic catch-all for all other errors
> >>
> >>  For validation reporting:
> >>  --What part of message failed validation
> >>  --What stellar validator caused the failure
> >>
> >>  ---
> >>  Thank you,
> >>
> >>  James Sirota
> >>  PPMC- Apache Metron (Incubating)
> >>  jsirota AT apache DOT org
> >>
> >>  --
> >>
> >>  Jon
> >>
> >>  Sent from my 

Re: [DISCUSS] Error Indexing

[James Sirota]

Jon,

I am still not entirely following why we would want to use hashing.  For 
example if my error is "Your IP field is invalid and failed validation" hashing 
this error string will always result in the same hash.  Why not just use the 
actual error string? Can you provide an example where you would use it?

Thanks,
James

23.01.2017, 16:29, "zeo...@gmail.com" :
> For 1 - I'm good with that.
>
> I'm talking about hashing the relevant content itself not the error. Some
> benefits are (1) minimize load on search index (there's minimal benefit in
> spending the CPU and disk to keep it at full fidelity (tokenize and store))
> (2) provide something to key on for dashboards (assuming a good hash
> algorithm that avoids collisions and is second preimage resistant) and (3)
> specific to errors, if the issue is that it failed to index, a hash gives
> us some protection that the issue will not occur twice.
>
> Jon
>
> On Mon, Jan 23, 2017, 2:47 PM James Sirota  wrote:
>
> Jon,
>
> With regards to 1, collapsing to a single dashboard for each would be
> fine. So we would have one error index and one "failed to validate"
> index. The distinction is that errors would be things that went wrong
> during stream processing (failed to parse, etc...), while validation
> failures are messages that explicitly failed stellar validation/schema
> enforcement. There should be relatively few of the second type.
>
> With respect to 3, why do you want the error hashed? Why not just search
> for the error text?
>
> Thanks,
> James
>
> 20.01.2017, 14:01, "zeo...@gmail.com" :
>>  As someone who currently fills the platform engineer role, I can give this
>>  idea a huge +1. My thoughts:
>>
>>  1. I think it depends on exactly what data is pushed into the index (#3).
>>  However, assuming the errors you proposed recording, I can't see huge
>>  benefits to having more than one dashboard. I would be happy to be
>>  persuaded otherwise.
>>
>>  2. I would say yes, storing the errors in HDFS in addition to indexing is
>>  a good thing. Using METRON-510
>>   as a case study, there
>>  is the potential in this environment for attacker-controlled data to
>
> result
>>  in processing errors which could be a method of evading security
>>  monitoring. Once an attack is identified, the long term HDFS storage would
>>  allow better historical analysis for low-and-slow/persistent attacks (I'm
>>  thinking of a method of data exfil that also won't successfully get stored
>>  in Lucene, but is hard to identify over a short period of time).
>>   - Along this line, I think that there are various parts of Metron (this
>>  included) which could benefit from having method of configuring data aging
>>  by bucket in HDFS (Following Nick's comments here
>>  ).
>>
>>  3. I would potentially add a hash of the content that failed validation to
>>  help identify repeats over time with less of a concern that you'd have
>
> back
>>  to back failures (i.e. instead of storing the value itself). Additionally,
>>  I think it's helpful to be able to search all times there was an indexing
>>  error (instead of it hitting the catch-all).
>>
>>  Jon
>>
>>  On Fri, Jan 20, 2017 at 1:17 PM James Sirota  wrote:
>>
>>  We already have a capability to capture bolt errors and validation errors
>>  and pipe them into a Kafka topic. I want to propose that we attach a
>>  writer topology to the error and validation failed kafka topics so that we
>>  can (a) create a new ES index for these errors and (b) create a new Kibana
>>  dashboard to visualize them. The benefit would be that errors and
>>  validation failures would be easier to see and analyze.
>>
>>  I am seeking feedback on the following:
>>
>>  - How granular would we want this feature to be? Think we would want one
>>  index/dashboard per source? Or would it be better to collapse everything
>>  into the same index?
>>  - Do we care about storing these errors in HDFS as well? Or is indexing
>>  them enough?
>>  - What types of errors should we record? I am proposing:
>>
>>  For error reporting:
>>  --Message failed to parse
>>  --Enrichment failed to enrich
>>  --Threat intel feed failures
>>  --Generic catch-all for all other errors
>>
>>  For validation reporting:
>>  --What part of message failed validation
>>  --What stellar validator caused the failure
>>
>>  ---
>>  Thank you,
>>
>>  James Sirota
>>  PPMC- Apache Metron (Incubating)
>>  jsirota AT apache DOT org
>>
>>  --
>>
>>  Jon
>>
>>  Sent from my mobile device
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
> --
>
> Jon
>
> Sent from my mobile device

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: [DISCUSS] Error Indexing

[zeo...@gmail.com]

For 1 - I'm good with that.

I'm talking about hashing the relevant content itself not the error.  Some
benefits are (1) minimize load on search index (there's minimal benefit in
spending the CPU and disk to keep it at full fidelity (tokenize and store))
(2) provide something to key on for dashboards (assuming a good hash
algorithm that avoids collisions and is second preimage resistant) and (3)
specific to errors, if the issue is that it failed to index, a hash gives
us some protection that the issue will not occur twice.

Jon

On Mon, Jan 23, 2017, 2:47 PM James Sirota  wrote:

Jon,

With regards to 1, collapsing to a single dashboard for each would be
fine.  So we would have one error index and one "failed to validate"
index.  The distinction is that errors would be things that went wrong
during stream processing (failed to parse, etc...), while validation
failures are messages that explicitly failed stellar validation/schema
enforcement.  There should be relatively few of the second type.


With respect to 3, why do you want the error hashed?  Why not just search
for the error text?

Thanks,
James


20.01.2017, 14:01, "zeo...@gmail.com" :
> As someone who currently fills the platform engineer role, I can give this
> idea a huge +1. My thoughts:
>
> 1. I think it depends on exactly what data is pushed into the index (#3).
> However, assuming the errors you proposed recording, I can't see huge
> benefits to having more than one dashboard. I would be happy to be
> persuaded otherwise.
>
> 2. I would say yes, storing the errors in HDFS in addition to indexing is
> a good thing. Using METRON-510
>  as a case study, there
> is the potential in this environment for attacker-controlled data to
result
> in processing errors which could be a method of evading security
> monitoring. Once an attack is identified, the long term HDFS storage would
> allow better historical analysis for low-and-slow/persistent attacks (I'm
> thinking of a method of data exfil that also won't successfully get stored
> in Lucene, but is hard to identify over a short period of time).
>  - Along this line, I think that there are various parts of Metron (this
> included) which could benefit from having method of configuring data aging
> by bucket in HDFS (Following Nick's comments here
> ).
>
> 3. I would potentially add a hash of the content that failed validation to
> help identify repeats over time with less of a concern that you'd have
back
> to back failures (i.e. instead of storing the value itself). Additionally,
> I think it's helpful to be able to search all times there was an indexing
> error (instead of it hitting the catch-all).
>
> Jon
>
> On Fri, Jan 20, 2017 at 1:17 PM James Sirota  wrote:
>
> We already have a capability to capture bolt errors and validation errors
> and pipe them into a Kafka topic. I want to propose that we attach a
> writer topology to the error and validation failed kafka topics so that we
> can (a) create a new ES index for these errors and (b) create a new Kibana
> dashboard to visualize them. The benefit would be that errors and
> validation failures would be easier to see and analyze.
>
> I am seeking feedback on the following:
>
> - How granular would we want this feature to be? Think we would want one
> index/dashboard per source? Or would it be better to collapse everything
> into the same index?
> - Do we care about storing these errors in HDFS as well? Or is indexing
> them enough?
> - What types of errors should we record? I am proposing:
>
> For error reporting:
> --Message failed to parse
> --Enrichment failed to enrich
> --Threat intel feed failures
> --Generic catch-all for all other errors
>
> For validation reporting:
> --What part of message failed validation
> --What stellar validator caused the failure
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
> --
>
> Jon
>
> Sent from my mobile device

---
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

-- 

Jon

Sent from my mobile device

[GitHub] incubator-metron issue #408: METRON-608 Mpack to install a single-node test ...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/408
  
@JonZeolla , I'm not sure I follow the doc statement, since the issue is 
not installing two MySQL instances on the same server, but rather sharing the 
same MySQL instance between two services (Hive and Metron).

At any rate, I think you are correct that this is an obsolete statement in 
the doc, since [METRON-410](https://issues.apache.org/jira/browse/METRON-410) 
\([#317](https://github.com/apache/incubator-metron/pull/317)\) was fixed in 
November.

Are you suggesting that since I'm in this doc anyway, I should fix this doc 
issue by removing that statement?  Okay by me, just clarifying.  Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #420: METRON-668: Remove the "tickUpdate" profile con...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/420
  
By the way, please don't merge this.  I'm using this branch to figure out 
what's going on with the build failures.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Build failing

[Casey Stella]

One more thing, just for posterity here, it always freezes at 6 records
written to HDFS.  That's the reason I thought it was a flushing issue.

On Mon, Jan 23, 2017 at 3:38 PM, Casey Stella  wrote:

> Ok, so now I'm concerned that this isn't a fluke.  Here's an excerpt from
> the failing logs on travis for my PR with substantially longer timeouts (
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/194575474/log.txt)
>
> Running org.apache.metron.solr.integration.SolrIndexingIntegrationTest
> 0 vs 10 vs 0
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed 
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 317.056 sec 
> <<< FAILURE!
> test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)  Time 
> elapsed: 316.949 sec  <<< ERROR!
> java.lang.RuntimeException: Took too long to complete: 300783 > 30
>   at 
> org.apache.metron.integration.ComponentRunner.process(ComponentRunner.java:131)
>   at 
> org.apache.metron.indexing.integration.IndexingIntegrationTest.test(IndexingIntegrationTest.java:173)
>   
>
> I'm getting the impression that this isn't the timeout and we have a mystery 
> on our hands.  Each of those lines "10 vs 10 vs 6" happen 15 seconds apart.  
> That line means that it read 10 entries from kafka, 10 entries from the 
> indexed data and 6 entries from HDFS.  It's that 6 entries that is the 
> problem.   Also of note, this does not seem to happen to me locally AND it's 
> not consistent on Travis.  Given all that I'd say that it's a problem with 
> the HDFS Writer not getting flushed, but I verified that it is indeed flushed 
> per message.
>
>
> Anyway, tl;dr we have a mystery unit test bug that isn't deterministic wrt 
> the unit tests and may or may not manifest itself outside of the unit tests.  
> So, yeah, I'll be looking at it, but would appreciate others taking a gander 
> too.
>
>
> Casey
>
>
> On Mon, Jan 23, 2017 at 2:09 PM, Casey Stella  wrote:
>
>> Yeah, I adjusted the timeout on the indexing integration tests as part of
>> https://github.com/apache/incubator-metron/pull/420 which I'll merge in
>> today.
>>
>> On Mon, Jan 23, 2017 at 2:01 PM, zeo...@gmail.com 
>> wrote:
>>
>>> Okay, now we have back to back failures, and it looks like it may have
>>> been
>>> a timeout issue?
>>>  `test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest):
>>> Took too long to complete: 150582 > 15`, more details below:
>>>
>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>>> 166.167 sec <<< FAILURE!
>>>
>>> test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)
>>> Time elapsed: 166.071 sec  <<< ERROR!
>>>
>>> java.lang.RuntimeException: Took too long to complete: 150582 > 15
>>>
>>> at org.apache.metron.integration.ComponentRunner.process(Compon
>>> entRunner.java:131)
>>>
>>> at org.apache.metron.indexing.integration.IndexingIntegrationTe
>>> st.test(IndexingInt

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

[James Sirota]

We used jollyday for OpenSOC for holiday resolution
http://jollyday.sourceforge.net/

I think its apache licensed. 

23.01.2017, 13:42, "Michael Miklavcic" :
> Casey,
>
> I think this is a move in the right direction. I am partial to the DSL.
> While it is another DSL to learn, I believe it is far easier to understand
> and write lookback rules for than it would be using the map of values.
>
> I'd like to suggest that the concept of holiday should be pluggable by the
> user. I would also like to make a case for the includes and excludes
> prioritization being based on order in the DSL. So in the holiday example
> you might be able to say something like PROFILE_LOOKBACK( '1 hour bins from
> 1 hour to 1 month including tuesdays excluding holidays including newyears')
>
> Thanks,
> Mike
>
> On Mon, Jan 23, 2017 at 1:01 PM, Casey Stella  wrote:
>
>>  Hi All,
>>
>>  I'm planning to expand the capabilities of PROFILE_GET and wanted to pass
>>  an idea past the community.
>>
>>  *Current State*
>>
>>  Currently, the functionality of PROFILE_GET is fairly straightforward:
>>
>> - profile - The name of the profile.
>> - entity - The name of the entity.
>> - durationAgo - How long ago should values be retrieved from?
>> - units - The units of 'durationAgo'.
>> - groups_list - Optional, must correspond to the 'groupBy' list used in
>> profile creation - List (in square brackets) of groupBy values used to
>> filter the profile. Default is the empty list, meaning groupBy was not
>>  used
>> when creating the profile.
>> - config_overrides - Optional - Map (in curly braces) of name:value
>> pairs, each overriding the global config parameter of the same name.
>> Default is the empty Map, meaning no overrides.
>>
>>  This has the advantage of providing a relatively simple mechanism to
>>  support the dominant use-case, gathering the profiles for a trailing
>>  window. The issues, however, are a couple:
>>
>> - We may need more complex semantics for specifying the window
>> (motivated below)
>> - As such, this couples the gathering of the profiles with the
>> specification of the window.
>>
>>  I propose to decouple these two concepts. I propose that we extract the
>>  notion of the lookback into a separate, more featureful function called
>>  PROFILE_LOOKBACK() which could be composed with an adjusted PROFILE_GET,
>>  whose arguments look like:
>>
>> - profile - The name of the profile.
>> - entity - The name of the entity.
>> - timestamps - The list of timestamps to retrieve
>> - groups_list - Optional, must correspond to the 'groupBy' list used in
>> profile creation - List (in square brackets) of groupBy values used to
>> filter the profile. Default is the empty list, meaning groupBy was not
>>  used
>> when creating the profile.
>> - config_overrides - Optional - Map (in curly braces) of name:value
>> pairs, each overriding the global config parameter of the same name.
>> Default is the empty Map, meaning no overrides.
>>
>>  So, PROFILE_GET would have the output of PROFILE_LOOKBACK passed to it as
>>  its 3rd argument (e.g. PROFILE_GET( 'my_profile', 'my_entity',
>>  PROFILE_LOOKBACK(...)) ).
>>
>>  *Motivation for Change*
>>
>>  The justification for this is that sometimes you want to compare time bins
>>  for a long duration back, but you don't want to skew the data by including
>>  periods that aren't distributionally similar (due to seasonal data, for
>>  instance). You might want to compare a value to statistically baseline of
>>  the median of the values for the same time window on the same day for the
>>  last month (e.g. every tuesday at this time).
>>
>>  Also, we might want a trailing window that does not start at the current
>>  time (in wall-clock), but rather starts an hour back or from the time that
>>  the data was originally ingested.
>>
>>  *PROFILE_LOOKBACK*
>>
>>  I propose that we support the following features:
>>
>> - A starting point that is not current time
>> - Sparse bins (i.e. the last hour for every tuesday for the last month)
>> - The ability to skip events (e.g. weekends, holidays)
>>
>>  This would result in a new function with the following arguments:
>>
>> -
>>
>> from - The lookback starting point (default to now)
>> -
>>
>> fromUnits - The units for the lookback starting point
>> -
>>
>> to - The ending point for the lookback window (default to from +
>>  binSize)
>> -
>>
>> toUnits - The units for the lookback ending point
>> -
>>
>> including - A list of conditions which we would skip.
>> - weekend
>>    - holiday
>>    - sunday through saturday
>> -
>>
>> excluding - A list of conditions which we would skip.
>> - weekend
>>    - holiday
>>    - sunday through saturday
>> -
>>
>> binSize - The size of the lookback bin
>> -
>>
>> binUnits - The units of the lookback bin
>>
>>  Given th

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

[James Sirota]

I am +1 on this feature.  It opens the door to true statistical baselining.  
The key motivating use case for me is as follows:

Lets say I am looking at the number of flows originating from my server A to 
external assets (anything not on my network) on a Tuesday at 1pm.  I want to 
figure out what number or range of A-> external flows constitutes normal.  I 
would query every bin on a Tuesday a 1pm for the last 5 Tuesdays, figure out 
25/50/75% values are for these bins and I would know (a) my 'normal' range and 
(b) if what I have currently is an anomaly.  

23.01.2017, 13:01, "Casey Stella" :
> Hi All,
>
> I'm planning to expand the capabilities of PROFILE_GET and wanted to pass
> an idea past the community.
>
> *Current State*
>
> Currently, the functionality of PROFILE_GET is fairly straightforward:
>
>    - profile - The name of the profile.
>    - entity - The name of the entity.
>    - durationAgo - How long ago should values be retrieved from?
>    - units - The units of 'durationAgo'.
>    - groups_list - Optional, must correspond to the 'groupBy' list used in
>    profile creation - List (in square brackets) of groupBy values used to
>    filter the profile. Default is the empty list, meaning groupBy was not used
>    when creating the profile.
>    - config_overrides - Optional - Map (in curly braces) of name:value
>    pairs, each overriding the global config parameter of the same name.
>    Default is the empty Map, meaning no overrides.
>
> This has the advantage of providing a relatively simple mechanism to
> support the dominant use-case, gathering the profiles for a trailing
> window. The issues, however, are a couple:
>
>    - We may need more complex semantics for specifying the window
>    (motivated below)
>    - As such, this couples the gathering of the profiles with the
>    specification of the window.
>
> I propose to decouple these two concepts. I propose that we extract the
> notion of the lookback into a separate, more featureful function called
> PROFILE_LOOKBACK() which could be composed with an adjusted PROFILE_GET,
> whose arguments look like:
>
>    - profile - The name of the profile.
>    - entity - The name of the entity.
>    - timestamps - The list of timestamps to retrieve
>    - groups_list - Optional, must correspond to the 'groupBy' list used in
>    profile creation - List (in square brackets) of groupBy values used to
>    filter the profile. Default is the empty list, meaning groupBy was not used
>    when creating the profile.
>    - config_overrides - Optional - Map (in curly braces) of name:value
>    pairs, each overriding the global config parameter of the same name.
>    Default is the empty Map, meaning no overrides.
>
> So, PROFILE_GET would have the output of PROFILE_LOOKBACK passed to it as
> its 3rd argument (e.g. PROFILE_GET( 'my_profile', 'my_entity',
> PROFILE_LOOKBACK(...)) ).
>
> *Motivation for Change*
>
> The justification for this is that sometimes you want to compare time bins
> for a long duration back, but you don't want to skew the data by including
> periods that aren't distributionally similar (due to seasonal data, for
> instance). You might want to compare a value to statistically baseline of
> the median of the values for the same time window on the same day for the
> last month (e.g. every tuesday at this time).
>
> Also, we might want a trailing window that does not start at the current
> time (in wall-clock), but rather starts an hour back or from the time that
> the data was originally ingested.
>
> *PROFILE_LOOKBACK*
>
> I propose that we support the following features:
>
>    - A starting point that is not current time
>    - Sparse bins (i.e. the last hour for every tuesday for the last month)
>    - The ability to skip events (e.g. weekends, holidays)
>
> This would result in a new function with the following arguments:
>
>    -
>
>    from - The lookback starting point (default to now)
>    -
>
>    fromUnits - The units for the lookback starting point
>    -
>
>    to - The ending point for the lookback window (default to from + binSize)
>    -
>
>    toUnits - The units for the lookback ending point
>    -
>
>    including - A list of conditions which we would skip.
>    - weekend
>   - holiday
>   - sunday through saturday
>    -
>
>    excluding - A list of conditions which we would skip.
>    - weekend
>   - holiday
>   - sunday through saturday
>    -
>
>    binSize - The size of the lookback bin
>    -
>
>    binUnits - The units of the lookback bin
>
> Given the number of arguments and their complexity and the fact that many,
> many are optional, I propose that either
>
>    - PROFILE_LOOKBACK take a Map so that we can get essentially named
>    params in stellar.
>    - PROFILE_LOOKBACK accept a string backed by a DSL to express these
>    criteria
>
> Ok, so that's a lot to take in. How about we look at some motivating
> use-cases.
>
> *Base Case: A lookback of 1 hour ago*
>
> As a map, this would

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

[Casey Stella]

Yeah, that 'holidays' bit is harder than I initially thought.

On Mon, Jan 23, 2017 at 3:42 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Casey,
>
> I think this is a move in the right direction. I am partial to the DSL.
> While it is another DSL to learn, I believe it is far easier to understand
> and write lookback rules for than it would be using the map of values.
>
> I'd like to suggest that the concept of holiday should be pluggable by the
> user. I would also like to make a case for the includes and excludes
> prioritization being based on order in the DSL. So in the holiday example
> you might be able to say something like PROFILE_LOOKBACK( '1 hour bins from
> 1 hour to 1 month including tuesdays excluding holidays including
> newyears')
>
> Thanks,
> Mike
>
>
> On Mon, Jan 23, 2017 at 1:01 PM, Casey Stella  wrote:
>
> > Hi All,
> >
> > I'm planning to expand the capabilities of PROFILE_GET and wanted to pass
> > an idea past the community.
> >
> > *Current State*
> >
> > Currently, the functionality of PROFILE_GET is fairly straightforward:
> >
> >- profile - The name of the profile.
> >- entity - The name of the entity.
> >- durationAgo - How long ago should values be retrieved from?
> >- units - The units of 'durationAgo'.
> >- groups_list - Optional, must correspond to the 'groupBy' list used
> in
> >profile creation - List (in square brackets) of groupBy values used to
> >filter the profile. Default is the empty list, meaning groupBy was not
> > used
> >when creating the profile.
> >- config_overrides - Optional - Map (in curly braces) of name:value
> >pairs, each overriding the global config parameter of the same name.
> >Default is the empty Map, meaning no overrides.
> >
> > This has the advantage of providing a relatively simple mechanism to
> > support the dominant use-case, gathering the profiles for a trailing
> > window.  The issues, however, are a couple:
> >
> >- We may need more complex semantics for specifying the window
> >(motivated below)
> >- As such, this couples the gathering of the profiles with the
> >specification of the window.
> >
> > I propose to decouple these two concepts. I propose that we extract the
> > notion of the lookback into a separate, more featureful function called
> > PROFILE_LOOKBACK() which could be composed with an adjusted PROFILE_GET,
> > whose arguments look like:
> >
> >
> >- profile - The name of the profile.
> >- entity - The name of the entity.
> >- timestamps - The list of timestamps to retrieve
> >- groups_list - Optional, must correspond to the 'groupBy' list used
> in
> >profile creation - List (in square brackets) of groupBy values used to
> >filter the profile. Default is the empty list, meaning groupBy was not
> > used
> >when creating the profile.
> >- config_overrides - Optional - Map (in curly braces) of name:value
> >pairs, each overriding the global config parameter of the same name.
> >Default is the empty Map, meaning no overrides.
> >
> > So, PROFILE_GET would have the output of PROFILE_LOOKBACK passed to it as
> > its 3rd argument (e.g. PROFILE_GET( 'my_profile', 'my_entity',
> > PROFILE_LOOKBACK(...)) ).
> >
> > *Motivation for Change*
> >
> > The justification for this is that sometimes you want to compare time
> bins
> > for a long duration back, but you don't want to skew the data by
> including
> > periods that aren't distributionally similar (due to seasonal data, for
> > instance).  You might want to compare a value to statistically baseline
> of
> > the median of the values for the same time window on the same day for the
> > last month (e.g. every tuesday at this time).
> >
> > Also, we might want a trailing window that does not start at the current
> > time (in wall-clock), but rather starts an hour back or from the time
> that
> > the data was originally ingested.
> >
> >
> > *PROFILE_LOOKBACK*
> >
> > I propose that we support the following features:
> >
> >- A starting point that is not current time
> >- Sparse bins (i.e. the last hour for every tuesday for the last
> month)
> >- The ability to skip events (e.g. weekends, holidays)
> >
> >
> > This would result in a new function with the following arguments:
> >
> >-
> >
> >from - The lookback starting point (default to now)
> >-
> >
> >fromUnits - The units for the lookback starting point
> >-
> >
> >to - The ending point for the lookback window (default to from +
> > binSize)
> >-
> >
> >toUnits - The units for the lookback ending point
> >-
> >
> >including - A list of conditions which we would skip.
> >- weekend
> >   - holiday
> >   - sunday through saturday
> >-
> >
> >excluding - A list of conditions which we would skip.
> >- weekend
> >   - holiday
> >   - sunday through saturday
> >-
> >
> >binSize - The size of the lookback bin
> >-
> >
> >binU

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

[Michael Miklavcic]

Casey,

I think this is a move in the right direction. I am partial to the DSL.
While it is another DSL to learn, I believe it is far easier to understand
and write lookback rules for than it would be using the map of values.

I'd like to suggest that the concept of holiday should be pluggable by the
user. I would also like to make a case for the includes and excludes
prioritization being based on order in the DSL. So in the holiday example
you might be able to say something like PROFILE_LOOKBACK( '1 hour bins from
1 hour to 1 month including tuesdays excluding holidays including newyears')

Thanks,
Mike


On Mon, Jan 23, 2017 at 1:01 PM, Casey Stella  wrote:

> Hi All,
>
> I'm planning to expand the capabilities of PROFILE_GET and wanted to pass
> an idea past the community.
>
> *Current State*
>
> Currently, the functionality of PROFILE_GET is fairly straightforward:
>
>- profile - The name of the profile.
>- entity - The name of the entity.
>- durationAgo - How long ago should values be retrieved from?
>- units - The units of 'durationAgo'.
>- groups_list - Optional, must correspond to the 'groupBy' list used in
>profile creation - List (in square brackets) of groupBy values used to
>filter the profile. Default is the empty list, meaning groupBy was not
> used
>when creating the profile.
>- config_overrides - Optional - Map (in curly braces) of name:value
>pairs, each overriding the global config parameter of the same name.
>Default is the empty Map, meaning no overrides.
>
> This has the advantage of providing a relatively simple mechanism to
> support the dominant use-case, gathering the profiles for a trailing
> window.  The issues, however, are a couple:
>
>- We may need more complex semantics for specifying the window
>(motivated below)
>- As such, this couples the gathering of the profiles with the
>specification of the window.
>
> I propose to decouple these two concepts. I propose that we extract the
> notion of the lookback into a separate, more featureful function called
> PROFILE_LOOKBACK() which could be composed with an adjusted PROFILE_GET,
> whose arguments look like:
>
>
>- profile - The name of the profile.
>- entity - The name of the entity.
>- timestamps - The list of timestamps to retrieve
>- groups_list - Optional, must correspond to the 'groupBy' list used in
>profile creation - List (in square brackets) of groupBy values used to
>filter the profile. Default is the empty list, meaning groupBy was not
> used
>when creating the profile.
>- config_overrides - Optional - Map (in curly braces) of name:value
>pairs, each overriding the global config parameter of the same name.
>Default is the empty Map, meaning no overrides.
>
> So, PROFILE_GET would have the output of PROFILE_LOOKBACK passed to it as
> its 3rd argument (e.g. PROFILE_GET( 'my_profile', 'my_entity',
> PROFILE_LOOKBACK(...)) ).
>
> *Motivation for Change*
>
> The justification for this is that sometimes you want to compare time bins
> for a long duration back, but you don't want to skew the data by including
> periods that aren't distributionally similar (due to seasonal data, for
> instance).  You might want to compare a value to statistically baseline of
> the median of the values for the same time window on the same day for the
> last month (e.g. every tuesday at this time).
>
> Also, we might want a trailing window that does not start at the current
> time (in wall-clock), but rather starts an hour back or from the time that
> the data was originally ingested.
>
>
> *PROFILE_LOOKBACK*
>
> I propose that we support the following features:
>
>- A starting point that is not current time
>- Sparse bins (i.e. the last hour for every tuesday for the last month)
>- The ability to skip events (e.g. weekends, holidays)
>
>
> This would result in a new function with the following arguments:
>
>-
>
>from - The lookback starting point (default to now)
>-
>
>fromUnits - The units for the lookback starting point
>-
>
>to - The ending point for the lookback window (default to from +
> binSize)
>-
>
>toUnits - The units for the lookback ending point
>-
>
>including - A list of conditions which we would skip.
>- weekend
>   - holiday
>   - sunday through saturday
>-
>
>excluding - A list of conditions which we would skip.
>- weekend
>   - holiday
>   - sunday through saturday
>-
>
>binSize - The size of the lookback bin
>-
>
>binUnits - The units of the lookback bin
>
> Given the number of arguments and their complexity and the fact that many,
> many are optional, I propose that either
>
>- PROFILE_LOOKBACK take a Map so that we can get essentially named
>params in stellar.
>- PROFILE_LOOKBACK accept a string backed by a DSL to express these
>criteria
>
>
> Ok, so that's a lot to take in.  How about we look at some motivati

Re: Build failing

[Casey Stella]

Ok, so now I'm concerned that this isn't a fluke.  Here's an excerpt from
the failing logs on travis for my PR with substantially longer timeouts (
https://s3.amazonaws.com/archive.travis-ci.org/jobs/194575474/log.txt)

Running org.apache.metron.solr.integration.SolrIndexingIntegrationTest
0 vs 10 vs 0
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Processed 
target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
10 vs 10 vs 6
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
317.056 sec <<< FAILURE!
test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)
Time elapsed: 316.949 sec  <<< ERROR!
java.lang.RuntimeException: Took too long to complete: 300783 > 30
at 
org.apache.metron.integration.ComponentRunner.process(ComponentRunner.java:131)
at 
org.apache.metron.indexing.integration.IndexingIntegrationTest.test(IndexingIntegrationTest.java:173)


I'm getting the impression that this isn't the timeout and we have a
mystery on our hands.  Each of those lines "10 vs 10 vs 6" happen 15
seconds apart.  That line means that it read 10 entries from kafka, 10
entries from the indexed data and 6 entries from HDFS.  It's that 6
entries that is the problem.   Also of note, this does not seem to
happen to me locally AND it's not consistent on Travis.  Given all
that I'd say that it's a problem with the HDFS Writer not getting
flushed, but I verified that it is indeed flushed per message.


Anyway, tl;dr we have a mystery unit test bug that isn't deterministic
wrt the unit tests and may or may not manifest itself outside of the
unit tests.  So, yeah, I'll be looking at it, but would appreciate
others taking a gander too.


Casey


On Mon, Jan 23, 2017 at 2:09 PM, Casey Stella  wrote:

> Yeah, I adjusted the timeout on the indexing integration tests as part of
> https://github.com/apache/incubator-metron/pull/420 which I'll merge in
> today.
>
> On Mon, Jan 23, 2017 at 2:01 PM, zeo...@gmail.com 
> wrote:
>
>> Okay, now we have back to back failures, and it looks like it may have
>> been
>> a timeout issue?
>>  `test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest):
>> Took too long to complete: 150582 > 15`, more details below:
>>
>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>> 166.167 sec <<< FAILURE!
>>
>> test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)
>> Time elapsed: 166.071 sec  <<< ERROR!
>>
>> java.lang.RuntimeException: Took too long to complete: 150582 > 15
>>
>> at org.apache.metron.integration.ComponentRunner.process(Compon
>> entRunner.java:131)
>>
>> at org.apache.metron.indexing.integration.IndexingIntegrationTe
>> st.test(IndexingIntegrationTest.java:173)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:62)
>>
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:483)
>>
>> at org.junit.runners.model.

[GitHub] incubator-metron issue #419: METRON-664: Make the index configuration per-wr...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/419
  
Tested this in quick-dev. +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[DISCUSS] Expansion of the capabilities of PROFILE_GET

[Casey Stella]

Hi All,

I'm planning to expand the capabilities of PROFILE_GET and wanted to pass
an idea past the community.

*Current State*

Currently, the functionality of PROFILE_GET is fairly straightforward:

   - profile - The name of the profile.
   - entity - The name of the entity.
   - durationAgo - How long ago should values be retrieved from?
   - units - The units of 'durationAgo'.
   - groups_list - Optional, must correspond to the 'groupBy' list used in
   profile creation - List (in square brackets) of groupBy values used to
   filter the profile. Default is the empty list, meaning groupBy was not used
   when creating the profile.
   - config_overrides - Optional - Map (in curly braces) of name:value
   pairs, each overriding the global config parameter of the same name.
   Default is the empty Map, meaning no overrides.

This has the advantage of providing a relatively simple mechanism to
support the dominant use-case, gathering the profiles for a trailing
window.  The issues, however, are a couple:

   - We may need more complex semantics for specifying the window
   (motivated below)
   - As such, this couples the gathering of the profiles with the
   specification of the window.

I propose to decouple these two concepts. I propose that we extract the
notion of the lookback into a separate, more featureful function called
PROFILE_LOOKBACK() which could be composed with an adjusted PROFILE_GET,
whose arguments look like:


   - profile - The name of the profile.
   - entity - The name of the entity.
   - timestamps - The list of timestamps to retrieve
   - groups_list - Optional, must correspond to the 'groupBy' list used in
   profile creation - List (in square brackets) of groupBy values used to
   filter the profile. Default is the empty list, meaning groupBy was not used
   when creating the profile.
   - config_overrides - Optional - Map (in curly braces) of name:value
   pairs, each overriding the global config parameter of the same name.
   Default is the empty Map, meaning no overrides.

So, PROFILE_GET would have the output of PROFILE_LOOKBACK passed to it as
its 3rd argument (e.g. PROFILE_GET( 'my_profile', 'my_entity',
PROFILE_LOOKBACK(...)) ).

*Motivation for Change*

The justification for this is that sometimes you want to compare time bins
for a long duration back, but you don't want to skew the data by including
periods that aren't distributionally similar (due to seasonal data, for
instance).  You might want to compare a value to statistically baseline of
the median of the values for the same time window on the same day for the
last month (e.g. every tuesday at this time).

Also, we might want a trailing window that does not start at the current
time (in wall-clock), but rather starts an hour back or from the time that
the data was originally ingested.


*PROFILE_LOOKBACK*

I propose that we support the following features:

   - A starting point that is not current time
   - Sparse bins (i.e. the last hour for every tuesday for the last month)
   - The ability to skip events (e.g. weekends, holidays)


This would result in a new function with the following arguments:

   -

   from - The lookback starting point (default to now)
   -

   fromUnits - The units for the lookback starting point
   -

   to - The ending point for the lookback window (default to from + binSize)
   -

   toUnits - The units for the lookback ending point
   -

   including - A list of conditions which we would skip.
   - weekend
  - holiday
  - sunday through saturday
   -

   excluding - A list of conditions which we would skip.
   - weekend
  - holiday
  - sunday through saturday
   -

   binSize - The size of the lookback bin
   -

   binUnits - The units of the lookback bin

Given the number of arguments and their complexity and the fact that many,
many are optional, I propose that either

   - PROFILE_LOOKBACK take a Map so that we can get essentially named
   params in stellar.
   - PROFILE_LOOKBACK accept a string backed by a DSL to express these
   criteria


Ok, so that's a lot to take in.  How about we look at some motivating
use-cases.

*Base Case: A lookback of 1 hour ago*

As a map, this would look like:

PROFILE_LOOKBACK( { 'binSize' : 1, 'binUnits' : 'HOURS' } )

As a DSL this would look like:
PROFILE_LOOKBACK( '1 hour bins from now')


*The same time window every tuesday for the last month starting one hour
ago*

Just to make this as clear as possible, if this is run at 3PM on Monday
January 23rd, 2017, it would include the following bins:

   - January 17th, 2PM - 3PM
   - January 10th, 2PM - 3PM
   - January 3rd, 2PM - 3PM
   - December 27th, 2PM - 3PM

As a map, this would look like:

PROFILE_LOOKBACK( { 'from' : 1, 'fromUnits' : 'HOURS', 'to' : 1, 'toUnits'
: 'MONTH', 'including' : [ 'tuesday' ], 'binSize' : 1, 'binUnits' : 'HOURS'
} )

As a DSL this would look like:
PROFILE_LOOKBACK( '1 hour bins from 1 hour to 1 month including tuesdays')

*The same time window every sund

Re: [DISCUSS] Error Indexing

[James Sirota]

Jon,

With regards to 1, collapsing to a single dashboard for each would be fine.  So 
we would have one error index and one "failed to validate" index.  The 
distinction is that errors would be things that went wrong during stream 
processing (failed to parse, etc...), while validation failures are messages 
that explicitly failed stellar validation/schema enforcement.  There should be 
relatively few of the second type.


With respect to 3, why do you want the error hashed?  Why not just search for 
the error text?

Thanks,
James 


20.01.2017, 14:01, "zeo...@gmail.com" :
> As someone who currently fills the platform engineer role, I can give this
> idea a huge +1. My thoughts:
>
> 1. I think it depends on exactly what data is pushed into the index (#3).
> However, assuming the errors you proposed recording, I can't see huge
> benefits to having more than one dashboard. I would be happy to be
> persuaded otherwise.
>
> 2. I would say yes, storing the errors in HDFS in addition to indexing is
> a good thing. Using METRON-510
>  as a case study, there
> is the potential in this environment for attacker-controlled data to result
> in processing errors which could be a method of evading security
> monitoring. Once an attack is identified, the long term HDFS storage would
> allow better historical analysis for low-and-slow/persistent attacks (I'm
> thinking of a method of data exfil that also won't successfully get stored
> in Lucene, but is hard to identify over a short period of time).
>  - Along this line, I think that there are various parts of Metron (this
> included) which could benefit from having method of configuring data aging
> by bucket in HDFS (Following Nick's comments here
> ).
>
> 3. I would potentially add a hash of the content that failed validation to
> help identify repeats over time with less of a concern that you'd have back
> to back failures (i.e. instead of storing the value itself). Additionally,
> I think it's helpful to be able to search all times there was an indexing
> error (instead of it hitting the catch-all).
>
> Jon
>
> On Fri, Jan 20, 2017 at 1:17 PM James Sirota  wrote:
>
> We already have a capability to capture bolt errors and validation errors
> and pipe them into a Kafka topic. I want to propose that we attach a
> writer topology to the error and validation failed kafka topics so that we
> can (a) create a new ES index for these errors and (b) create a new Kibana
> dashboard to visualize them. The benefit would be that errors and
> validation failures would be easier to see and analyze.
>
> I am seeking feedback on the following:
>
> - How granular would we want this feature to be? Think we would want one
> index/dashboard per source? Or would it be better to collapse everything
> into the same index?
> - Do we care about storing these errors in HDFS as well? Or is indexing
> them enough?
> - What types of errors should we record? I am proposing:
>
> For error reporting:
> --Message failed to parse
> --Enrichment failed to enrich
> --Threat intel feed failures
> --Generic catch-all for all other errors
>
> For validation reporting:
> --What part of message failed validation
> --What stellar validator caused the failure
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
> --
>
> Jon
>
> Sent from my mobile device

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

[GitHub] incubator-metron issue #316: METRON-503: Metron REST API

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/316
  
@jjmeyer0, just pushed out the commit to separate Services into interfaces 
and implementations.  As I created the interfaces there were a few spots that 
needed some changes.  

First there were several methods in the GrokService that didn't belong 
there.  These include methods that support writing grok statements to/from HDFS 
and updating a sensor parser config with these statements.  I believe these 
methods should be in SensorParserConfigService since they are needed to support 
writing and reading sensor parser configs.  It should be noted that these 
methods can all be removed if we ever move to keeping grok statements in 
zookeeper instead of HDFS (see PR #308).

The second is something you alluded to earlier about converting the 
StormService to a facade pattern.  I think you are correct there, this service 
contains 2 different types of Storm interactions:

- function calls to the Storm REST API
- functions that are not exposed through the Storm REST API, only through 
the Storm CLI

I broke this service down into 2 smaller services.  Is that along the lines 
of what you were thinking?

I also moved all the constants in various classes to a single constant 
class.  We don't have that many right now so it's manageable to keep them in 
one place.  

Was also thinking we should rename TransformationService to StellarService? 
 I can see that evolving to be more generic than just the Stellar hook in the 
transformation phase of the parser topology.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #420: METRON-668: Remove the "tickUpdate" profile con...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/420
  
The explicit test that was done on quickdev was following the Median 
Absolute Deviation scenario in `metron-statistics`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Build failing

[Casey Stella]

Yeah, I adjusted the timeout on the indexing integration tests as part of
https://github.com/apache/incubator-metron/pull/420 which I'll merge in
today.

On Mon, Jan 23, 2017 at 2:01 PM, zeo...@gmail.com  wrote:

> Okay, now we have back to back failures, and it looks like it may have been
> a timeout issue?
>  `test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest):
> Took too long to complete: 150582 > 15`, more details below:
>
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 166.167 sec <<< FAILURE!
>
> test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)
> Time elapsed: 166.071 sec  <<< ERROR!
>
> java.lang.RuntimeException: Took too long to complete: 150582 > 15
>
> at org.apache.metron.integration.ComponentRunner.process(
> ComponentRunner.java:131)
>
> at org.apache.metron.indexing.integration.
> IndexingIntegrationTest.test(IndexingIntegrationTest.java:173)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:483)
>
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> FrameworkMethod.java:50)
>
> at org.junit.internal.runners.model.ReflectiveCallable.run(
> ReflectiveCallable.java:12)
>
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(
> FrameworkMethod.java:47)
>
> at org.junit.internal.runners.statements.InvokeMethod.
> evaluate(InvokeMethod.java:17)
>
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(
> BlockJUnit4ClassRunner.java:78)
>
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(
> BlockJUnit4ClassRunner.java:57)
>
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>
> at org.junit.runners.ParentRunner.runChildren(
> ParentRunner.java:288)
>
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>
> at org.junit.runners.ParentRunner$2.evaluate(
> ParentRunner.java:268)
>
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>
> at org.apache.maven.surefire.junit4.JUnit4Provider.execute(
> JUnit4Provider.java:252)
>
> at org.apache.maven.surefire.junit4.JUnit4Provider.
> executeTestSet(JUnit4Provider.java:141)
>
> at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(
> JUnit4Provider.java:112)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:483)
>
> at org.apache.maven.surefire.util.ReflectionUtils.
> invokeMethodWithArray(ReflectionUtils.java:189)
>
> at org.apache.maven.surefire.booter.ProviderFactory$
> ProviderProxy.invoke(ProviderFactory.java:165)
>
> at org.apache.maven.surefire.booter.ProviderFactory.
> invokeProvider(ProviderFactory.java:85)
>
> at org.apache.maven.surefire.booter.ForkedBooter.
> runSuitesInProcess(ForkedBooter.java:115)
>
> at org.apache.maven.surefire.booter.ForkedBooter.main(
> ForkedBooter.java:75)
>
>
> Jon
>
> On Thu, Jan 19, 2017 at 2:49 PM zeo...@gmail.com  wrote:
>
> > The build has been showing as failing
> >  for a little while now.  I
> > know we recently updated the language around Merge Requirements
> >  action?pageId=61332235>,
> > but if I recall properly our current issue is simply a Travis CI overload
> > issue.  Is there a way we can update the wiki doc to account for
> situations
> > like this?
> >
> > Jon
> > --
> >
> > Jon
> >
> > Sent from my mobile device
> >
> --
>
> Jon
>
> Sent from my mobile device
>

[GitHub] incubator-metron issue #420: METRON-668: Remove the "tickUpdate" profile con...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/420
  
Ok, test is complete, I made sure the profiler worked.  I also adjusted the 
timeout on the indexing integration tests so that travis doesn't barf.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Build failing

[zeo...@gmail.com]

Okay, now we have back to back failures, and it looks like it may have been
a timeout issue?
 `test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest):
Took too long to complete: 150582 > 15`, more details below:

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
166.167 sec <<< FAILURE!

test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)
Time elapsed: 166.071 sec  <<< ERROR!

java.lang.RuntimeException: Took too long to complete: 150582 > 15

at 
org.apache.metron.integration.ComponentRunner.process(ComponentRunner.java:131)

at 
org.apache.metron.indexing.integration.IndexingIntegrationTest.test(IndexingIntegrationTest.java:173)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:483)

at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)

at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)

at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)

at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)

at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)

at org.junit.runners.ParentRunner.run(ParentRunner.java:363)

at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)

at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)

at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:483)

at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)

at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)

at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)

at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)

at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)


Jon

On Thu, Jan 19, 2017 at 2:49 PM zeo...@gmail.com  wrote:

> The build has been showing as failing
>  for a little while now.  I
> know we recently updated the language around Merge Requirements
> ,
> but if I recall properly our current issue is simply a Travis CI overload
> issue.  Is there a way we can update the wiki doc to account for situations
> like this?
>
> Jon
> --
>
> Jon
>
> Sent from my mobile device
>
-- 

Jon

Sent from my mobile device

Re: [DISCUSS] Error Indexing

[James Sirota]

Thanks for your feedback, Jon.  Anyone else has interest/feedback for this 
feature? 

20.01.2017, 14:01, "zeo...@gmail.com" :
> As someone who currently fills the platform engineer role, I can give this
> idea a huge +1. My thoughts:
>
> 1. I think it depends on exactly what data is pushed into the index (#3).
> However, assuming the errors you proposed recording, I can't see huge
> benefits to having more than one dashboard. I would be happy to be
> persuaded otherwise.
>
> 2. I would say yes, storing the errors in HDFS in addition to indexing is
> a good thing. Using METRON-510
>  as a case study, there
> is the potential in this environment for attacker-controlled data to result
> in processing errors which could be a method of evading security
> monitoring. Once an attack is identified, the long term HDFS storage would
> allow better historical analysis for low-and-slow/persistent attacks (I'm
> thinking of a method of data exfil that also won't successfully get stored
> in Lucene, but is hard to identify over a short period of time).
>  - Along this line, I think that there are various parts of Metron (this
> included) which could benefit from having method of configuring data aging
> by bucket in HDFS (Following Nick's comments here
> ).
>
> 3. I would potentially add a hash of the content that failed validation to
> help identify repeats over time with less of a concern that you'd have back
> to back failures (i.e. instead of storing the value itself). Additionally,
> I think it's helpful to be able to search all times there was an indexing
> error (instead of it hitting the catch-all).
>
> Jon
>
> On Fri, Jan 20, 2017 at 1:17 PM James Sirota  wrote:
>
> We already have a capability to capture bolt errors and validation errors
> and pipe them into a Kafka topic. I want to propose that we attach a
> writer topology to the error and validation failed kafka topics so that we
> can (a) create a new ES index for these errors and (b) create a new Kibana
> dashboard to visualize them. The benefit would be that errors and
> validation failures would be easier to see and analyze.
>
> I am seeking feedback on the following:
>
> - How granular would we want this feature to be? Think we would want one
> index/dashboard per source? Or would it be better to collapse everything
> into the same index?
> - Do we care about storing these errors in HDFS as well? Or is indexing
> them enough?
> - What types of errors should we record? I am proposing:
>
> For error reporting:
> --Message failed to parse
> --Enrichment failed to enrich
> --Threat intel feed failures
> --Generic catch-all for all other errors
>
> For validation reporting:
> --What part of message failed validation
> --What stellar validator caused the failure
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
> --
>
> Jon
>
> Sent from my mobile device

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: [VOTE] Release Process Documentation

[James Sirota]

Sorry that's a typo. Was meant to be 10.  Just fixed

20.01.2017, 13:22, "zeo...@gmail.com" :
> -1 (non-binding).
>
> There appears to be a minor oversight where it goes from Step 9 to Step 14.
>
> Jon
>
> On Fri, Jan 20, 2017 at 11:56 AM James Sirota  wrote:
>
>>  The document is available here:
>>  https://cwiki.apache.org/confluence/display/METRON/Release+Process
>>
>>  and is also pasted in this email for your convenience
>>
>>  Please vote +1, -1, or 0 for neutral. The vote will be open for 72 hours
>>
>>  Metron Release Types
>>  There are two types of Metron releases:
>>  Feature Release (FR) - this is a release that has a significant step
>>  forward in feature capability and is denoted by an upgrade of the second
>>  digit
>>  Maintenance Release (MR) - this is a set of patches and fixes that are
>>  issued following the FR and is denoted by an upgrade of the third digit
>>  Release Naming Convention
>>  Metron build naming convention is as follows: 0.[FR].[MR]. We keep the 0.
>>  notation to signify that the project is still under active development and
>>  we will hold a community vote to go to 1.x at a future time
>>  Initiating a New Metron Release
>>  Create the MR branch for the previous Metron release by incrementing the
>>  *third* digit of the previous release like so 0.[FR].[*MR++*]. All patches
>>  to the previous Metron release will be checked in under the MR branch and
>>  where it makes sense also under the FR branch. All new features will be
>>  checked in under the FR branch.
>>  Creating a Feature Release
>>  Step 1 - Initiate a discuss thread
>>  Prior to the release The Release manager should do the following
>>  (preferably a month before the release):
>>  Make sure that the list of JIRAs slated for the release accurately
>>  reflects to reflects the pull requests that are currently in master
>>  Construct an email to the Metron dev board (
>>  dev@metron.incubator.apache.org) which discusses with the community the
>>  desire to do a release. This email should contain the following:
>>  The list of JIRAs slated for the release with descriptions (use the output
>>  of git log and remove all the JIRAs from the last release’s changelog)
>>  A solicitation of JIRAs that should be included with the next release.
>>  Users should rate them as must/need/good to have as well as volunteering.
>>  A release email template is provided here.
>>  Step 2 - Monitor and Verify JIRAs
>>  Once the community votes for additional JIRAs they want included in the
>>  release verify that the pull requests are in before the release, close
>>  these JIRAs and tag them with the release name. All pull requests and JIRAs
>>  that were not slated for this release will go into the next releases. The
>>  release manager should continue to monitor the JIRA to ensure that the
>>  timetable is on track until the release date. On the release date the
>>  release manager should message the Metron dev board (
>>  dev@metron.incubator.apache.org) announcing the code freeze for the
>>  release.
>>  Step 3 - Create the Release Branch and Increment Metron version
>>  Create an branch for the release (from a repo cloned from
>>  https://git-wip-us.apache.org/repos/asf/incubator-metron.git). (assuming
>>  the release is 0.[FR++].0 and working from master):
>>  git checkout -b Metron_0.[FR++].0
>>  git push --set-upstream origin Metron_0.[FR++].0
>>  File a JIRA to increment the Metron version to 0.[FR++].0. Either do it
>>  yourself or have a community member increment the build version for you.
>>  You can look at a pull request for a previous build to see how this is
>>  done. METRON-533 - Up the version for release DONE
>>  Also, the release manager should have a couple of things set up:
>>  A SVN clone of the repo at
>>  https://dist.apache.org/repos/dist/dev/incubator/metron, We will refer to
>>  this as the dev repo. It will hold the release candidate artifacts
>>  A SVN clone of the repo at
>>  https://dist.apache.org/repos/dist/release/incubator/metron, We will
>>  refer to this as the release repo. It will hold the release artifacts.
>>  Step 4 - Create the Release Candidate
>>
>>  Now, for each release candidate, we will tag from that branch. Assuming
>>  that this is RC1:
>>  git checkout Metron_0.[FR++].0 && git pull
>>  git tag apache-metron-0.[FR++].0-rc1-incubating
>>  git push origin —tags
>>  Now we must create the release candidate tarball. From the apache repo,
>>  you should run:
>>
>>   git archive --prefix=apache-metron-0.[FR++].0-rc1-incubating/
>>   apache-metron-0.[FR++].0-rc1-incubating | gzip >
>>   apache-metron-0.[FR++].0-rc-incubating.tar.gz
>>
>>  We will refer to this as the release candidate tarball. *Note: Per Apache
>>  policy, the hardware used to create the candidate tarball must be owned by
>>  the release manager.
>>  The artifacts for a release (or a release candidate, for that matter) are
>>  as follows:
>>  Release (candidate) Tarball
>>   MD5 hash of the release tarball. We wil

Re: [DISCUSS] Mtron Project Maturity

[James Sirota]

Ok revised.  Anyone else wanted to chime in?
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66852119

Code
CD10The project produces Open Source software, for distribution to the public 
at no charge. 
All of Metron source code is Open Source under the Apache 2.0 license.  We do 
not charge for the software
1CD20The project's code is easily discoverable and publicly accessible.
All of Metron source code is staged in Github at the following URL: 
https://github.com/apache/incubator-metron.  Metron also has a website at the 
following URL: http://metron.incubator.apache.org/.  A Google search for 
"Apache Metron" brings up these links #4 and #1 on the list respectively. 
CD30The code can be built in a reproducible way using widely available standard 
tools.
All of Metron source code (not inclusive of build scripts and configuration 
files)  can be built via Maven from a top-level POM file
CD40The full history of the project's code is available via a source code 
control system, in a way that allows any released version to be recreated.
We maintain release branches per every Metron release in source control.  The 
list of release branches can be referenced here: 
https://github.com/apache/incubator-metron/releases
CD50The provenance of each line of code is established via the source code 
control system, in a reliable way based on strong authentication of the 
committer. When third-party contributions are committed, commit messages 
provide reliable information about the code provenance. 2
Full attribution to the committer is provided upon the merge of the pull 
request.  This is recorded in two places.  First, it is recorded when a pull 
request is committed.  Second, this is recorded in the JIRA workflow.

Licenses and Copyright
LC10The code is released under the Apache License, version 2.0.
All Metron code is released under the Apache 2.0 License.  We have had 4 
sanctioned Apache releases that are staged here: 
https://dist.apache.org/repos/dist/release/incubator/metron/
LC20Libraries that are mandatory dependencies of the project's code do not 
create more restrictions than the Apache License does. 3 4
Every direct or transient dependency packaged with Metron adheres to an 
Apache-compliant license.  All dependencies with non-compliant licenses were 
removed
LC30The libraries mentioned in LC20 are available as Open Source software.
As far as we are aware all libraries mentioned in LC20 are available as Open 
Source software.  We (a) use Maven to pull these libraries from an on-line 
source so we are sure they are freely available for download and (b) we use a 
combination of Apache RAT and a Checkstyle plug-in to check for license 
compliance. 
LC40Committers are bound by an Individual Contributor Agreement (the "Apache 
iCLA") that defines which code they are allowed to commit and how they need to 
identify code that is not their own.
All Metron committers submitted a signed individual ICLA
LC50The copyright ownership of everything that the project produces is clearly 
defined and documented. 5
We document copyrights of all included source code in compliance with Apache 
licensing standards.
Releases
RE10Releases consist of source code, distributed using standard and open 
archive formats that are expected to stay readable in the long term. 6
Metron is released using the .tar.gz format
RE20Releases are approved by the project's PMC (see CS10), in order to make 
them an act of the Foundation.
Each release is voted on by the PPMC members, who contribute binding votes to 
make the release.
RE30Releases are signed and/or distributed along with digests that can be 
reliably used to validate the downloaded archives.
Links to the releases are provided along with hashes and signatures of the 
release to validate its authenticity
RE40Convenience binaries can be distributed alongside source code but they are 
not Apache Releases -- they are just a convenience provided with no guarantee.
Metron does not distribute convenience binaries
RE50The release process is documented and repeatable to the extent that someone 
new to the project is able to independently generate the complete set of 
artifacts required for a release.
Our release process is documented here: Release Process
Quality
QU10The project is open and honest about the quality of its code. Various 
levels of quality and maturity for various modules are natural and acceptable 
as long as they are clearly communicated.
Metron provides instructions on how to validate and test as a part of every 
voting email.  Further more, there is a integration and unit test framework 
that runs during every Maven build to catch bugs at build time.  
Instructions on how to verify builds are provided here: Verifying Builds
An email template announcing the build and providing a link to verification 
instructions can be found here: Build Vote Template
QU20The project puts a very high priority on producing secure software. 7
As documented in our release instruc

[GitHub] incubator-metron issue #420: METRON-668: Remove the "tickUpdate" profile con...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/420
  
@nickwallen running as we speak.  The test plan will be to generate some 
data like we do for the MAD outliers and construct a couple simple profiles.  
I'll make it a bit more explicit after I do it. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #417: METRON-659 Emulate Sensors in Developmen...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/417


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #420: METRON-668: Remove the "tickUpdate" profile con...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/420
  
I like this as it simplifies the Profile configuration.

How did you test this?  Did you run any profiles on "Quick Dev"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #422: METRON-670 Monit Incorrectly Reports Sta...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/incubator-metron/pull/422

METRON-670 Monit Incorrectly Reports Status

In a constrained environment, like 'Quick Dev', Monit will often 
incorrectly report the status of a Metron topology. This occurs when the 
environment is under load and a query of topology status exceeds the default 
timeout of 30 seconds. 

Added a parameter so that the timeout for a status check can be extended 
under these conditions. This was previously done for starting and stopping a 
topology, but not for a status checks.

This was tested in 'Quick Dev' and made starting, stopping and reporting 
status of the topologies using Monit work much better.  Previously Monit would 
erroneously report some of the topologies as not running when they were.  This 
would also interfere with your ability to start/stop the same topologies.  

For example, starting all of the services required to consume Bro telemetry 
works much better with this change.
```
monit -g bro start
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-670

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #422


commit adf86d7f8c536d26e9d078768ad8e158fe33bbf2
Author: Nick Allen 
Date:   2017-01-23T14:51:27Z

METRON-670 Monit Incorrectly Reports Status




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/421#discussion_r97334025
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBolt.java
 ---
@@ -161,6 +167,7 @@ protected void initializeStellar() {
 stellarContext = new Context.Builder()
  .with(Context.Capabilities.ZOOKEEPER_CLIENT, () 
-> client)
  .with(Context.Capabilities.GLOBAL_CONFIG, () -> 
getConfigurations().getGlobalConfig())
+ .with(Context.Capabilities.GEO_IP, () -> 
GeoLiteDatabase.INSTANCE)
--- End diff --

Also- why is this in the GenericEnrichment bolt? Am I mistaken, or wouldn't 
this be better located in the GEO bolt?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/421#discussion_r97327302
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBolt.java
 ---
@@ -161,6 +167,7 @@ protected void initializeStellar() {
 stellarContext = new Context.Builder()
  .with(Context.Capabilities.ZOOKEEPER_CLIENT, () 
-> client)
  .with(Context.Capabilities.GLOBAL_CONFIG, () -> 
getConfigurations().getGlobalConfig())
+ .with(Context.Capabilities.GEO_IP, () -> 
GeoLiteDatabase.INSTANCE)
--- End diff --

Since the INSTANCE is static, is there a reason to not just refer to it 
directly in the Stellar function rather than adding a capability?  I always 
envisioned capabilities to be for configs which were changing at runtime.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/421#discussion_r97328119
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java
 ---
@@ -63,12 +65,15 @@ public ThreatIntelJoinBolt(String zookeeperUrl) {
   @Override
   public void prepare(Map map, TopologyContext topologyContext) {
 super.prepare(map, topologyContext);
+
GeoLiteDatabase.INSTANCE.update((String)getConfigurations().getGlobalConfig().get(GeoLiteDatabase.GEO_HDFS_FILE));
--- End diff --

This can certainly stand now, but it opens the question as to whether we 
want to register some easier way to do this than remembering to call the right 
set of functions in all of the bolts that use stellar as a follow-on JIRA.  
Just a thought.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/421#discussion_r97326393
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
 ---
@@ -225,6 +163,8 @@ indexing.executors=0
 kafka.zk={{ zookeeper_quorum }}
 kafka.broker={{ kafka_brokers }}
 kafka.start=WHERE_I_LEFT_OFF
+# GEO #
+geo.hdfs.file={{ geoip_hdfs_file }}
--- End diff --

Why is the geo IP data file in HDFS configured in flux rather than 
zookeeper?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/421#discussion_r97326080
  
--- Diff: LICENSE ---
@@ -210,6 +210,12 @@ This product bundles some test examples from the Stix 
project (metron-platform/m
 
 This product bundles wait-for-it.sh, which is available under a "MIT 
Software License" license.  For details, see 
https://github.com/vishnubob/wait-for-it
 

+
+
--- End diff --

Could you make the license explicit here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/421#discussion_r97327030
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
 ---
@@ -81,7 +81,6 @@ public JSONObject enrich(CacheKey value) {
 Map globalConfig = 
value.getConfig().getConfiguration();
 Map sensorConfig = 
value.getConfig().getEnrichment().getConfig();
 if(handler == null) {
-_LOG.trace("Stellar ConfigHandler is null.");
--- End diff --

Didn't like the trace statements there?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---