Re: [DISCUSS] Upcoming Release

[Ryan Merriman]

Makes sense now.  Thanks Matt.

> On Nov 17, 2017, at 4:25 PM, Matt Foley  wrote:
> 
> Hi Ryan,
> Yes and no.  The last release (see 
> https://dist.apache.org/repos/dist/release/metron/ ) was 0.4.1, announced on 
> 9/19.
> Immediately after that we bumped the  of builds from master branch, 
> per https://issues.apache.org/jira/browse/METRON-1196 .  This is consistent 
> with the Release Process “Clean up” phase: “It is good practice to increment 
> the build version in master immediately after a Feature Release, so that dev 
> builds with new stuff from master cannot be mistaken for builds of the 
> release version. So, immediately after a release, increment the MINOR version 
> number (eg, with the 0.4.0 just released, set the new version number to 
> 0.4.1)” 
> (https://cwiki.apache.org/confluence/display/METRON/Release+Process#ReleaseProcess-Step15-Cleanup
>  )
> 
> So you’re correct that the version of development builds is currently set to 
> 0.4.2.  After we actually make a release of 0.4.2, we’ll change dev build 
>  to 0.4.3 (regardless of whether we expect the following release to 
> be 0.4.3 or 0.5.0).
> 
> Hope this clarifies,
> --Matt
> 
> On 11/17/17, 1:59 PM, "Ryan Merriman"  wrote:
> 
>Matt,
> 
>I think we are currently on version 0.4.2.  If that is the case would the
>next version be 0.4.3?
> 
>Ryan
> 
>>On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley  wrote:
>> 
>> (With release manager hat on)
>> 
>> The community has proposed a release of Metron in the near future,
>> focusing on Meta-alerts running in Elasticsearch.
>> Congrats on getting so many of the below already done.  At this point,
>> only METRON-1252, and the discussion of how to handle joint release of the
>> Metron bro plugin, remain as gating items for the release.  I project these
>> will be resolved next week, so let’s propose the following:
>> 
>> Sometime next week, after the last bits are done, I’ll start the release
>> process and create the release branch.
>> 
>> The proposed new version will be 0.4.2, unless there are backward
>> incompatible changes that support making it 0.5.0.
>> Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
>> having Docs Text indicating so.
>> ***If anyone knows that some of the commits included since 0.4.1 introduce
>> backward incompatibility, please say so now on this thread, and mark the
>> Jira as such.***
>> 
>> The 90 or so jiras/commits already in master branch since 0.4.1 are listed
>> below.
>> Thanks,
>> --Matt
>> 
>>METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
>> Some Records (nickwallen) closes apache/metron#832
>>METRON-1294 IP addresses are not formatted correctly in facet and
>> group results (merrimanr) closes apache/metron#827
>>METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
>> cluster (merrimanr) closes apache/metron#826
>>METRON-1290 Only first 10 alerts are update when a MetaAlert status is
>> changed to inactive (justinleet) closes apache/metron#842
>>METRON-1311 Service Check Should Check Elasticsearch Index Templates
>> (nickwallen) closes apache/metron#839
>>METRON-1289 Alert fields are lost when a MetaAlert is created
>> (merrimanr) closes apache/metron#824
>>METRON-1309 Change metron-deployment to pull the plugin from
>> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
>>METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
>> closes apache/metron#838
>>METRON-1275: Fix Metron Documentation closes
>> apache/incubator-metron#833
>>METRON-1295 Unable to Configure Logging for REST API (nickwallen)
>> closes apache/metron#828
>>METRON-1307 Force install of java8 since java9 does not appear to work
>> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
>>METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
>> cestella) closes apache/incubator-metron#829
>>METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
>> closes apache/metron#821
>>METRON-1287 Full Dev Fails When Installing EPEL Repository
>> (nickwallen) closes apache/metron#820
>>METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
>> page (iraghumitra via merrimanr) closes apache/metron#819
>>METRON-1283 Install Elasticsearch template as a part of the mpack
>> startup scripts (anandsubbu via nickwallen) closes apache/metron#817
>>METRON-1254: Conditionals as map keys do not function in Stellar
>> closes apache/incubator-metron#801
>>METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
>> closes apache/metron#805
>>METRON-1284 Remove extraneous dead query in ElasticsearchDao
>> (justinleet) closes apache/metron#818
>>METRON-1270: fix for warnings missing @return tag argument in
>> metron-analytics/metron-profiler-common and metron-profiler-client closes
>> 

[GitHub] metron-bro-plugin-kafka pull request #3: METRON-813: Migrate metron-bro-plug...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron-bro-plugin-kafka/pull/3


---

[GitHub] metron-bro-plugin-kafka issue #3: METRON-813: Migrate metron-bro-plugin-kafk...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/3
  
+1 This works great @JonZeolla .  This is a really nice enhancement.  I 
tested by manually installing Bro and using bro-pkg to install our plugin.  
Everything works exactly as I would expect.


---

Re: [DISCUSS] Upcoming Release

[Matt Foley]

Hi Ryan,
Yes and no.  The last release (see 
https://dist.apache.org/repos/dist/release/metron/ ) was 0.4.1, announced on 
9/19.
Immediately after that we bumped the  of builds from master branch, 
per https://issues.apache.org/jira/browse/METRON-1196 .  This is consistent 
with the Release Process “Clean up” phase: “It is good practice to increment 
the build version in master immediately after a Feature Release, so that dev 
builds with new stuff from master cannot be mistaken for builds of the release 
version. So, immediately after a release, increment the MINOR version number 
(eg, with the 0.4.0 just released, set the new version number to 0.4.1)” 
(https://cwiki.apache.org/confluence/display/METRON/Release+Process#ReleaseProcess-Step15-Cleanup
 )

So you’re correct that the version of development builds is currently set to 
0.4.2.  After we actually make a release of 0.4.2, we’ll change dev build 
 to 0.4.3 (regardless of whether we expect the following release to be 
0.4.3 or 0.5.0).

Hope this clarifies,
--Matt

On 11/17/17, 1:59 PM, "Ryan Merriman"  wrote:

Matt,

I think we are currently on version 0.4.2.  If that is the case would the
next version be 0.4.3?

Ryan

On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley  wrote:

> (With release manager hat on)
>
> The community has proposed a release of Metron in the near future,
> focusing on Meta-alerts running in Elasticsearch.
> Congrats on getting so many of the below already done.  At this point,
> only METRON-1252, and the discussion of how to handle joint release of the
> Metron bro plugin, remain as gating items for the release.  I project 
these
> will be resolved next week, so let’s propose the following:
>
> Sometime next week, after the last bits are done, I’ll start the release
> process and create the release branch.
>
> The proposed new version will be 0.4.2, unless there are backward
> incompatible changes that support making it 0.5.0.
> Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
> having Docs Text indicating so.
> ***If anyone knows that some of the commits included since 0.4.1 introduce
> backward incompatibility, please say so now on this thread, and mark the
> Jira as such.***
>
> The 90 or so jiras/commits already in master branch since 0.4.1 are listed
> below.
> Thanks,
> --Matt
>
> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> Some Records (nickwallen) closes apache/metron#832
> METRON-1294 IP addresses are not formatted correctly in facet and
> group results (merrimanr) closes apache/metron#827
> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> cluster (merrimanr) closes apache/metron#826
> METRON-1290 Only first 10 alerts are update when a MetaAlert status is
> changed to inactive (justinleet) closes apache/metron#842
> METRON-1311 Service Check Should Check Elasticsearch Index Templates
> (nickwallen) closes apache/metron#839
> METRON-1289 Alert fields are lost when a MetaAlert is created
> (merrimanr) closes apache/metron#824
> METRON-1309 Change metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
> closes apache/metron#838
> METRON-1275: Fix Metron Documentation closes
> apache/incubator-metron#833
> METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> closes apache/metron#828
> METRON-1307 Force install of java8 since java9 does not appear to work
> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
> METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
> cestella) closes apache/incubator-metron#829
> METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
> closes apache/metron#821
> METRON-1287 Full Dev Fails When Installing EPEL Repository
> (nickwallen) closes apache/metron#820
> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> page (iraghumitra via merrimanr) closes apache/metron#819
> METRON-1283 Install Elasticsearch template as a part of the mpack
> startup scripts (anandsubbu via nickwallen) closes apache/metron#817
> METRON-1254: Conditionals as map keys do not function in Stellar
> closes apache/incubator-metron#801
> METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
> closes apache/metron#805
> METRON-1284 Remove extraneous dead query in ElasticsearchDao
> (justinleet) closes apache/metron#818
> METRON-1270: fix for warnings missing @return tag argument in
> 

Re: [DISCUSS] Upcoming Release

[Nick Allen]

Our last release was 0.4.1, so the next would be at least 0.4.2.  We
recently have been keeping master at the next presumed release version.

On Fri, Nov 17, 2017 at 4:59 PM, Ryan Merriman  wrote:

> Matt,
>
> I think we are currently on version 0.4.2.  If that is the case would the
> next version be 0.4.3?
>
> Ryan
>
> On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley  wrote:
>
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done.  At this point,
> > only METRON-1252, and the discussion of how to handle joint release of
> the
> > Metron bro plugin, remain as gating items for the release.  I project
> these
> > will be resolved next week, so let’s propose the following:
> >
> > Sometime next week, after the last bits are done, I’ll start the release
> > process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled ‘backward-incompatible’,
> nor
> > having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> introduce
> > backward incompatibility, please say so now on this thread, and mark the
> > Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1 are
> listed
> > below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> > Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet and
> > group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> > cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert status
> is
> > changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index Templates
> > (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> (nickwallen)
> > closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> > closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not appear to
> work
> > with the scripts (brianhurley via ottobackwards) closes apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
> > cestella) closes apache/incubator-metron#829
> > METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
> > closes apache/metron#821
> > METRON-1287 Full Dev Fails When Installing EPEL Repository
> > (nickwallen) closes apache/metron#820
> > METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> > page (iraghumitra via merrimanr) closes apache/metron#819
> > METRON-1283 Install Elasticsearch template as a part of the mpack
> > startup scripts (anandsubbu via nickwallen) closes apache/metron#817
> > METRON-1254: Conditionals as map keys do not function in Stellar
> > closes apache/incubator-metron#801
> > METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
> > closes apache/metron#805
> > METRON-1284 Remove extraneous dead query in ElasticsearchDao
> > (justinleet) closes apache/metron#818
> > METRON-1270: fix for warnings missing @return tag argument in
> > metron-analytics/metron-profiler-common and metron-profiler-client
> closes
> > apache/incubator-metron#810
> > METRON-1272 Hide child alerts from searches and grouping if they
> > belong to meta alerts (justinleet) closes apache/metron#811
> > METRON-1224 Add time range selection to search control (iraghumitra
> > via james-sirota) closes apache/metron#796
> > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
> > justinleet) closes apache/metron#816
> > METRON-1243: Add a REST endpoint which allows us to get a list of all
> > indice closes apache/incubator-metron#797
> > METRON-1196 Increment master version number to 0.4.2 for on-going
> > development (mattf-horton) closes apache/metron#767
> > METRON-1278 Strip Build Status widget from root README.md
> > in site-book build (mattf-horton) closes apache/metron#815
> > METRON-1274 Master has failure in StormControllerIntegrationTest
> > (merrimanr) closes apache/metron#813
> > METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes

[GitHub] metron pull request #843: METRON-1319: Column Metadata REST service should u...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/843#discussion_r151801296
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/SearchControllerIntegrationTest.java
 ---
@@ -132,6 +132,30 @@ public void testDefaultQuery() throws Exception {
 sensorIndexingConfigService.delete("bro");
   }
 
+  @Test
+  public void testDefaultColumnMetadata() throws Exception {
--- End diff --

Let me explain better.  I assume this is testing that the set of default 
indices are used.  That's just not clear to me how we are verifying that here.

Whereas, in your other test case 
`getColumnMetadataShouldProperlyGetDefaultIndices` that makes it very clear to 
me what you are testing. :)


---

[GitHub] metron pull request #843: METRON-1319: Column Metadata REST service should u...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/843#discussion_r151799957
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/SearchControllerIntegrationTest.java
 ---
@@ -132,6 +132,30 @@ public void testDefaultQuery() throws Exception {
 sensorIndexingConfigService.delete("bro");
   }
 
+  @Test
+  public void testDefaultColumnMetadata() throws Exception {
--- End diff --

Would you mind adding a comment somewhere that describes in more detail 
what this test case is testing?  Just looking at this test case in isolation, 
it is not clear to me what we are testing.


---

[GitHub] metron pull request #843: METRON-1319: Column Metadata REST service should u...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/843#discussion_r151798907
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/SearchServiceImpl.java
 ---
@@ -96,6 +96,11 @@ public GroupResponse group(GroupRequest groupRequest) 
throws RestException {
   @Override
   public Map getColumnMetadata(List indices) 
throws RestException {
 try {
+  if (indices == null || indices.isEmpty()) {
+indices = getDefaultIndices();
+// metaalerts should be included by default in column metadata 
requests
+indices.add(METAALERT_TYPE);
--- End diff --

Why is the metaalert index added in most cases, except in the case of 
group?  


---

[GitHub] metron pull request #843: METRON-1319: Column Metadata REST service should u...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/843#discussion_r151799462
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/SearchServiceImpl.java
 ---
@@ -96,6 +96,11 @@ public GroupResponse group(GroupRequest groupRequest) 
throws RestException {
   @Override
   public Map getColumnMetadata(List indices) 
throws RestException {
 try {
+  if (indices == null || indices.isEmpty()) {
+indices = getDefaultIndices();
--- End diff --

If you're motivated, it would make sense to add a log statement whenever we 
muck with the indices behind the scenes.  I feel like I'll be chasing a bug 
someday and not realize that the service layer is doing this.


---

Re: [DISCUSS] Upcoming Release

[Ryan Merriman]

Matt,

I think we are currently on version 0.4.2.  If that is the case would the
next version be 0.4.3?

Ryan

On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley  wrote:

> (With release manager hat on)
>
> The community has proposed a release of Metron in the near future,
> focusing on Meta-alerts running in Elasticsearch.
> Congrats on getting so many of the below already done.  At this point,
> only METRON-1252, and the discussion of how to handle joint release of the
> Metron bro plugin, remain as gating items for the release.  I project these
> will be resolved next week, so let’s propose the following:
>
> Sometime next week, after the last bits are done, I’ll start the release
> process and create the release branch.
>
> The proposed new version will be 0.4.2, unless there are backward
> incompatible changes that support making it 0.5.0.
> Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
> having Docs Text indicating so.
> ***If anyone knows that some of the commits included since 0.4.1 introduce
> backward incompatibility, please say so now on this thread, and mark the
> Jira as such.***
>
> The 90 or so jiras/commits already in master branch since 0.4.1 are listed
> below.
> Thanks,
> --Matt
>
> METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
> Some Records (nickwallen) closes apache/metron#832
> METRON-1294 IP addresses are not formatted correctly in facet and
> group results (merrimanr) closes apache/metron#827
> METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
> cluster (merrimanr) closes apache/metron#826
> METRON-1290 Only first 10 alerts are update when a MetaAlert status is
> changed to inactive (justinleet) closes apache/metron#842
> METRON-1311 Service Check Should Check Elasticsearch Index Templates
> (nickwallen) closes apache/metron#839
> METRON-1289 Alert fields are lost when a MetaAlert is created
> (merrimanr) closes apache/metron#824
> METRON-1309 Change metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
> closes apache/metron#838
> METRON-1275: Fix Metron Documentation closes
> apache/incubator-metron#833
> METRON-1295 Unable to Configure Logging for REST API (nickwallen)
> closes apache/metron#828
> METRON-1307 Force install of java8 since java9 does not appear to work
> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
> METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
> cestella) closes apache/incubator-metron#829
> METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
> closes apache/metron#821
> METRON-1287 Full Dev Fails When Installing EPEL Repository
> (nickwallen) closes apache/metron#820
> METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
> page (iraghumitra via merrimanr) closes apache/metron#819
> METRON-1283 Install Elasticsearch template as a part of the mpack
> startup scripts (anandsubbu via nickwallen) closes apache/metron#817
> METRON-1254: Conditionals as map keys do not function in Stellar
> closes apache/incubator-metron#801
> METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
> closes apache/metron#805
> METRON-1284 Remove extraneous dead query in ElasticsearchDao
> (justinleet) closes apache/metron#818
> METRON-1270: fix for warnings missing @return tag argument in
> metron-analytics/metron-profiler-common and metron-profiler-client closes
> apache/incubator-metron#810
> METRON-1272 Hide child alerts from searches and grouping if they
> belong to meta alerts (justinleet) closes apache/metron#811
> METRON-1224 Add time range selection to search control (iraghumitra
> via james-sirota) closes apache/metron#796
> METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
> justinleet) closes apache/metron#816
> METRON-1243: Add a REST endpoint which allows us to get a list of all
> indice closes apache/incubator-metron#797
> METRON-1196 Increment master version number to 0.4.2 for on-going
> development (mattf-horton) closes apache/metron#767
> METRON-1278 Strip Build Status widget from root README.md
> in site-book build (mattf-horton) closes apache/metron#815
> METRON-1274 Master has failure in StormControllerIntegrationTest
> (merrimanr) closes apache/metron#813
> METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
> apache/metron#809
> METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen)
> closes apache/metron#804
> METRON-1251: Typo and formatting fixes for metron-rest README closes
> apache/incubator-metron#800
> METRON-1241: Enable the REST API to use a cache for the zookeeper
> config similar to the Bolts closes apache/incubator-metron#795
> METRON-1267 Alerts UI returns a 404 when 

[GitHub] metron pull request #843: METRON-1319: Column Metadata REST service should u...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/843

METRON-1319: Column Metadata REST service should use default indices on 
empty input

## Contributor Comments
This PR adjusts the Column Metadata REST service to use a list of default 
indices when an empty list is passed in.  This behavior is similar to the 
search method and keeps the Alerts UI from having to track the current list of 
available indices.

I added test cases and verified the relevant tests in our Alerts UI 
end-to-end test now pass.  Still need to spin up full dev and validate there.  
Will report back once I have successfully done that but this can be reviewed in 
the meantime.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1319

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/843.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #843


commit ac4d68eea31caa1fa1a8f2f97c4d21256d7220e8
Author: merrimanr 
Date:   2017-11-17T21:36:00Z

initial commit




---

[GitHub] metron-bro-plugin-kafka pull request #2: DO NOT MERGE METRON-1304: Allow met...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r151794206
  
--- Diff: scripts/init.bro ---
@@ -18,11 +18,20 @@
 module Kafka;
 
 export {
-  const topic_name: string = "bro" 
-  const max_wait_on_shutdown: count = 3000 
-  const tag_json: bool = F 
-  const kafka_conf: table[string] of string = table(
-["metadata.broker.list"] = "localhost:9092"
-  ) 
-  const debug: string = "" 
+   ## Destination kafka topic name
+   const topic_name: string = "bro" 
+
+   ## Maximum wait on shutdown in milliseconds
+   const max_wait_on_shutdown: count = 3000 
+
+   ## Boolean to JSON with a log stream identifier
--- End diff --

Thanks for adding all the comments.But this one doesn't read so well.  
Or maybe I just don't read so well. :)  Read this one over once more.  If it 
still makes sense to you, then I'm good with it.


---

[GitHub] metron-bro-plugin-kafka pull request #2: DO NOT MERGE METRON-1304: Allow met...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r151793309
  
--- Diff: scripts/Bro/Kafka/logs-to-kafka.bro ---
@@ -14,32 +14,37 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-##! load this script to enable log output to kafka
+
+##! Load this script to enable log output to kafka
 
 module Kafka;
 
 export {
+   ## Specify which :bro:type:`Log::ID` to exclude from being sent to 
kafka.
##
-   ## which log streams should be sent to kafka?
-   ## example:
-   ##  redef Kafka::logs_to_send = set(Conn::Log, HTTP::LOG, 
DNS::LOG);
+   ## Example:  redef Kafka::logs_to_exclude = set(SSH::LOG);
+   const logs_to_exclude: set[Log::ID] 
+
+   ## Specify which :bro:type:`Log::ID` to send to kafka.
##
+   ## Example:  redef Kafka::logs_to_send = set(Conn::Log, DNS::LOG);
const logs_to_send: set[Log::ID] 
 }
 
 event bro_init() =-5
 {
for (stream_id in Log::active_streams)
{
-   if (stream_id in Kafka::logs_to_send)
-   {
-   local filter: Log::Filter = [
-   $name = fmt("kafka-%s", stream_id),
-   $writer = Log::WRITER_KAFKAWRITER,
-   $config = table(["stream_id"] = fmt("%s", 
stream_id))
-   ];
+   if ( stream_id in Kafka::logs_to_exclude ||
+   (|Kafka::logs_to_send| > 0 && stream_id !in 
Kafka::logs_to_send) )
--- End diff --

If a user configures nothing, neither defines `logs_to_send` nor 
`logs_to_exclude`, then we expect ALL logs to go to Kafka?


---

[GitHub] metron-bro-plugin-kafka pull request #2: DO NOT MERGE METRON-1304: Allow met...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r151791732
  
--- Diff: scripts/Bro/Kafka/logs-to-kafka.bro ---
@@ -14,32 +14,37 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-##! load this script to enable log output to kafka
+
+##! Load this script to enable log output to kafka
 
 module Kafka;
 
 export {
+   ## Specify which :bro:type:`Log::ID` to exclude from being sent to 
kafka.
##
-   ## which log streams should be sent to kafka?
-   ## example:
-   ##  redef Kafka::logs_to_send = set(Conn::Log, HTTP::LOG, 
DNS::LOG);
+   ## Example:  redef Kafka::logs_to_exclude = set(SSH::LOG);
+   const logs_to_exclude: set[Log::ID] 
+
+   ## Specify which :bro:type:`Log::ID` to send to kafka.
##
+   ## Example:  redef Kafka::logs_to_send = set(Conn::Log, DNS::LOG);
const logs_to_send: set[Log::ID] 
 }
 
 event bro_init() =-5
 {
for (stream_id in Log::active_streams)
{
-   if (stream_id in Kafka::logs_to_send)
-   {
-   local filter: Log::Filter = [
-   $name = fmt("kafka-%s", stream_id),
-   $writer = Log::WRITER_KAFKAWRITER,
-   $config = table(["stream_id"] = fmt("%s", 
stream_id))
-   ];
+   if ( stream_id in Kafka::logs_to_exclude ||
+   (|Kafka::logs_to_send| > 0 && stream_id !in 
Kafka::logs_to_send) )
--- End diff --

Why do we have to check that `logs_to_send` > 0 ? Is this necessary before 
doing a 'contains' (`in`)?

If it is necessary then we should do the same for `logs_to_exclude`.  If it 
is NOT necessary, then let's just get rid of it to simplify the logic.


---

Re: [DISCUSS] Upcoming Release

[Matt Foley]

(With release manager hat on)

The community has proposed a release of Metron in the near future, focusing on 
Meta-alerts running in Elasticsearch.
Congrats on getting so many of the below already done.  At this point, only 
METRON-1252, and the discussion of how to handle joint release of the Metron 
bro plugin, remain as gating items for the release.  I project these will be 
resolved next week, so let’s propose the following:

Sometime next week, after the last bits are done, I’ll start the release 
process and create the release branch.

The proposed new version will be 0.4.2, unless there are backward incompatible 
changes that support making it 0.5.0.
Currently there are NO included Jiras labeled ‘backward-incompatible’, nor 
having Docs Text indicating so.
***If anyone knows that some of the commits included since 0.4.1 introduce 
backward incompatibility, please say so now on this thread, and mark the Jira 
as such.***

The 90 or so jiras/commits already in master branch since 0.4.1 are listed 
below.
Thanks,
--Matt

METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters Some 
Records (nickwallen) closes apache/metron#832
METRON-1294 IP addresses are not formatted correctly in facet and group 
results (merrimanr) closes apache/metron#827
METRON-1291 Kafka produce REST endpoint does not work in a Kerberized 
cluster (merrimanr) closes apache/metron#826
METRON-1290 Only first 10 alerts are update when a MetaAlert status is 
changed to inactive (justinleet) closes apache/metron#842
METRON-1311 Service Check Should Check Elasticsearch Index Templates 
(nickwallen) closes apache/metron#839
METRON-1289 Alert fields are lost when a MetaAlert is created (merrimanr) 
closes apache/metron#824
METRON-1309 Change metron-deployment to pull the plugin from 
apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
METRON-1310 Template Delete Action Deletes Search Indices (nickwallen) 
closes apache/metron#838
METRON-1275: Fix Metron Documentation closes apache/incubator-metron#833
METRON-1295 Unable to Configure Logging for REST API (nickwallen) closes 
apache/metron#828
METRON-1307 Force install of java8 since java9 does not appear to work with 
the scripts (brianhurley via ottobackwards) closes apache/metron#835
METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via 
cestella) closes apache/incubator-metron#829
METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr) closes 
apache/metron#821
METRON-1287 Full Dev Fails When Installing EPEL Repository (nickwallen) 
closes apache/metron#820
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page 
(iraghumitra via merrimanr) closes apache/metron#819
METRON-1283 Install Elasticsearch template as a part of the mpack startup 
scripts (anandsubbu via nickwallen) closes apache/metron#817
METRON-1254: Conditionals as map keys do not function in Stellar closes 
apache/incubator-metron#801
METRON-1261 Apply bro security patch (JonZeolla via ottobackwards) closes 
apache/metron#805
METRON-1284 Remove extraneous dead query in ElasticsearchDao (justinleet) 
closes apache/metron#818
METRON-1270: fix for warnings missing @return tag argument in 
metron-analytics/metron-profiler-common and metron-profiler-client closes 
apache/incubator-metron#810
METRON-1272 Hide child alerts from searches and grouping if they belong to 
meta alerts (justinleet) closes apache/metron#811
METRON-1224 Add time range selection to search control (iraghumitra via 
james-sirota) closes apache/metron#796
METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via 
justinleet) closes apache/metron#816
METRON-1243: Add a REST endpoint which allows us to get a list of all 
indice closes apache/incubator-metron#797
METRON-1196 Increment master version number to 0.4.2 for on-going 
development (mattf-horton) closes apache/metron#767
METRON-1278 Strip Build Status widget from root README.md in 
site-book build (mattf-horton) closes apache/metron#815
METRON-1274 Master has failure in StormControllerIntegrationTest 
(merrimanr) closes apache/metron#813
METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes 
apache/metron#809
METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen) closes 
apache/metron#804
METRON-1251: Typo and formatting fixes for metron-rest README closes 
apache/incubator-metron#800
METRON-1241: Enable the REST API to use a cache for the zookeeper config 
similar to the Bolts closes apache/incubator-metron#795
METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page 
(merrimanr) closes apache/metron#808
METRON-1262 Unable to add comment for a alert in a meta-alert (merrimanr) 
closes apache/metron#806
METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via 
nickwallen) closes apache/metron#807
METRON-1255 MetaAlert 

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/803
  
Hi @iraghumitra - I'd like to see your work get in ASAP.  Can you merge 
with master when you get a chance?  

Also, I think you need to make some updates based on recent PRs that have 
gone in.  @justinleet pointed out that at a minimum the UI has to change for 
#824.  There may be others too.




---

Re: [DISCUSS] Upcoming Release

[Nick Allen]

I just wanted to send an update on where we are at.  We've gotten a lot
done here recently as you can see below.

  ✓ DONE (1) First, METRON-1289 needs to go in.  This one was a fairly big
effort and I am hearing that we are pretty close.

  ✓ DONE (2) METRON-1294 fixes an issue in how field types are looked-up.

  ✓ DONE (3) METRON-1290 is next.  While this may have been fixed in
M-1289, there may be some test cases we want from this PR.

  ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.

  ✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.

  (6) That leads us to Raghu's UI work in METRON-1252.  This introduces the
UI bits that depend on all the previous backend work.

  (7) At this point, we should have our best effort at running Metaalerts
on Elasticsearch 2.x. I propose that we cut a release here.

  (8) After we cut the release, we can introduce the work for ES 5.x in
METRON-939.  I know we will need lots of help testing and reviewing this
one.



We also have an outstanding question that needs resolved BEFORE we
release.  We need to come to a consensus on how to release having moved our
Bro Plugin to a separate repo.  I don't think we've heard from everyone on
this.  I'd urge everyone to chime in so we can choose a path forward.

If anyone is totally confused in regards to that discussion, I can try and
send an options summary again as a separate discuss thread.  The original
chain was somewhere around here [1].

[1]
https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E



On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen  wrote:

> Hi Guys -
>
> I want to follow-up on this discussion.  It sounds like most people are in
> agreement with the general approach.
>
> A lot of people have been working hard on Metaalerts and Elasticsearch.  I
> have checked-in with those doing the heavy lifting and have compiled a more
> detailed plan based on where we are at now.  To the best of my knowledge
> here is the plan of attack for finishing out this effort.
>
>   (1) First, METRON-1289 needs to go in.  This one was a fairly big effort
> and I am hearing that we are pretty close.
>
>   (2) METRON-1294 fixes an issue in how field types are looked-up.
>
>   (3) METRON-1290 is next.  While this may have been fixed in M-1289,
> there may be some test cases we want from this PR.
>
>   (4) METRON-1301 addresses a problem with the sorting logic.
>
>   (5) METRON-1291 fixes an issue with escalation of metaalerts.
>
>   (6) That leads us to Raghu's UI work in METRON-1252.  This introduces
> the UI bits that depend on all the previous backend work.
>
>   (7) At this point, we should have our best effort at running Metaalerts
> on Elasticsearch 2.x. I propose that we cut a release here.
>
>   (8) After we cut the release, we can introduce the work for ES 5.x in
> METRON-939.  I know we will need lots of help testing and reviewing this
> one.
>
> Please correct me if I am wrong.  I will try and send out updates as we
> make progress.
>
>
>
>
>
> On Mon, Nov 6, 2017 at 1:03 PM, zeo...@gmail.com  wrote:
>
>> I agree, I think it's very reasonable to move in line with Nick's
>> proposal.  I would also suggest that we outline what the target versions
>> would be to add in the METRON-777 components, since it has been functional
>> for a very long time but not reviewed and has some really rockstar
>> improvements.
>>
>> Jon
>>
>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler 
>> wrote:
>>
>> > I think the ES cutover should be the start of the 0.5.x series, and we
>> > continue on with 0.4.x for the
>> > metadata improvements etc.  We could chose to focus 0.5.x’s first
>> releases
>> > on not only ES but
>> > getting a handle on kibana and the mpack situation as well.
>> >
>> >
>> >
>> >
>> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>> > michael.miklav...@gmail.com) wrote:
>> >
>> > I agree with your proposal, Nick. I think having a stabilizing release
>> > prior to upgrading ES/Kibana makes sense.
>> >
>> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen  wrote:
>> >
>> > > I would like to start a discussion around upcoming releases. We have a
>> > > couple separate significant tracks of work that we need to reconcile
>> in
>> > our
>> > > release schedule.
>> > >
>> > > (1) We have had (and have in review) a good number of bug fixes
>> required
>> > to
>> > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
>> > >
>> > >
>> > > (2) We also have ongoing work to upgrade our infrastructure to
>> > > Elasticsearch 5.x, which will not be backwards compatible.
>> > >
>> > >
>> > > I would like to see a release that has our best work on ES 2.x before
>> we
>> > > migrate to 5.x. I would propose the following.
>> > >
>> > > Release N+1: Introduce Metaalerts running on ES 2.x
>> > >
>> > > Release N+2: Cut-over to ES 5.x
>> > >
>> > >

[GitHub] metron pull request #832: METRON-1301 Sorting on Triage Score Unexpectedly F...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/832


---

[GitHub] metron issue #832: METRON-1301 Sorting on Triage Score Unexpectedly Filters ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/832
  
Thanks for all the reviews, guys.  Going to commit now.


---

[GitHub] metron issue #832: METRON-1301 Sorting on Triage Score Unexpectedly Filters ...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/832
  
+1 worked as advertised.  Thanks @nickwallen!


---

[GitHub] metron issue #832: METRON-1301 Sorting on Triage Score Unexpectedly Filters ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/832
  
+1 by inspection.  Looks great!


---

[GitHub] metron issue #832: METRON-1301 Sorting on Triage Score Unexpectedly Filters ...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/832
  
Spun this up on full dev, and saw that sorting worked as expected in the 
UI.  This worked both when no groupings were selected and also when drilling 
down (e.g. grouping by ip_dst_addr and country).

I also like the refactorings a lot.  It's good stuff that makes life easier 
and more testable.

+1, assuming @cestella is happy with the changes you made on his comments.


---

[GitHub] metron pull request #827: METRON-1294: IP addresses are not formatted correc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/827


---

[GitHub] metron issue #827: METRON-1294: IP addresses are not formatted correctly in ...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/827
  
Great, thanks for the update.  +1


---

[GitHub] metron issue #827: METRON-1294: IP addresses are not formatted correctly in ...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/827
  
@nickwallen The parameter name is preexisting and I'm fine with leaving 
that.  Would you be okay with changing the descriptions in the README and 
annotations?  That should be a nonintrusive change that doesn't require 
spinning everything back up.

@merrimanr Do you have any objections to doing that as a middle ground?



---

[GitHub] metron issue #827: METRON-1294: IP addresses are not formatted correctly in ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/827
  
@justinleet I think that's a good find.  I'd suggest we fix this issue on a 
subsequent PR after we get this and #832 merged.  On #832, the refactoring will 
make it much easier to unit test a fix for this condition.  Let me know if you 
think that works or if you think its critical on this PR.


---

[GitHub] metron issue #827: METRON-1294: IP addresses are not formatted correctly in ...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/827
  
I ran a request giving sensors:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '["snort", "bro"]' 
'http://node1:8082/api/v1/search/column/metadata'
```
which returns fine
```
{
  "TTLs": "double",
  "bro_timestamp": "string",
  "enrichments:geo:ip_dst_addr:location_point": "other",
  "sha256": "string",
  "enrichmentjoinbolt:joiner:ts": "date",
  "certificate:version": "integer",
...
}
```

but giving the actual indices returns nothing, e.g.
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '["bro_index_2017.11.17.14"]' 
'http://node1:8082/api/v1/search/column/metadata'
```

I assume it's intentional that indices don't actually return data, which 
I'm fine with, but we need to rename things from indices to sensor or something.


---

[GitHub] metron pull request #826: METRON-1291: Kafka produce REST endpoint does not ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/826


---