Re: [DISCUSS] Next Release (0.3.1) Content

[zeo...@gmail.com]

I haven't had a chance to look through the unresolved JIRAs but I did want
to mention a few quick things.

First, when we released 0.3.0 and dropped the BETA flag, one of the things
that was discussed was putting together a method of documenting upgrades
from one version to the next.  As one of the first steps toward making that
a reasonable process, I think we should assemble more detailed release
notes, especially outlining non-backwards compatible changes.  In the
"[DISCUSS] Next Release Name" email thread Kyle Richardson suggested we use
"UPGRADING.md" to do this, and I still agree with that thought, but I'm
open to alternatives.

Separately, a *nice to have* would be *METRON-660*, which was discussed in
the "[PROPOSAL] up-to-date versioned documentation" thread, to give us some
cleaner documentation using the existing READMEs.  I'd be happy to help
with this one, I'm just not sure what the next steps are, aside from the
start that Matt has here
.

My last *nice to have* is *METRON-635*, which I have a *PR open for here
*.  If I could get
someone else to reproduce the error that I'm seeing I would be happy to
pursue additional testing, troubleshooting, etc.  I've seen others report
the same issue

on the HCC boards, so I'm fairly confident that it is not an issue with my
local environment.

Jon

On Thu, Jan 26, 2017 at 6:18 PM Casey Stella  wrote:

I took the liberty of adding the pull requests since the new year into the
in-progress list in the previous email.  If you own one of these and do not
believe that you can complete the review in the next week or so, let me
know and we can remove.  The only one of these that I see as mandatory is
METRON-650 because the reliance on the opensoc github repo might cause
issues with the release being acceptable (see discussion by Mike Miklavcic
with the mentors a couple weeks ago).

On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella  wrote:

> Hello Everyone!
>
> It's been almost 2 months since the last major release and I think it
> might be time to do a minor release of 0.3.1. The purpose of this email is
> multifold:
>
>- to take stock of what we have already committed
>- figure out what (if anything) is missing to make a release that
>we're proud of
>- find volunteers for the missing items
>
> For those who get email alerts on the JIRA changes, it should be no
> surprise that I have gone through and put the committed items in the 0.3.1
> release bucket.  So, let's take stock of what we have.
>
> *What's made it so far into the next release*
>
>
>
>- METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet)
>closes apache/incubator-metron#421
>- METRON-668: Remove the "tickUpdate" profile config and make the
>"init" phase not reset variables closes apache/incubator-metron#420
>- METRON-672: SolrIndexingIntegrationTest fails intermittently closes
>apache/incubator-metron#424
>- METRON-664: Make the index configuration per-writer with
>enabled/disabled closes apache/incubator-metron#419
>- METRON-600: Fix Metron Website (iraghumitra via cestella) closes
>apache/incubator-metron#399
>- METRON-666 Fix javadoc doclint errors closes
>apache/incubator-metron#418
>- METRON-659 Emulate Sensors in Development Environments (nickwallen)
>closes apache/incubator-metron#417
>- METRON-652: Extract indexing config from enrichment config closes
>apache/incubator-metron#415
>- METRON-654 Create RPM Installer for Profiler (nickwallen) closes
>apache/incubator-metron#413
>- METRON-656: Make Stellar 'in' closer to functioning like python
>closes apache/incubator-metron#416
>- METRON-532 Define Profile Period When Calling PROFILE_GET closes
>apache/incubator-metron#414
>- METRON-624: Updated Comparison/Equality Evaluations in Stellar
>closes apache/incubator-metron#404
>- METRON-644 RPM builds only work with Docker for Mac (kylerichardson)
>closes apache/incubator-metron#409
>- METRON-640: Add a Stellar function to compute shannon entropy for
>strings closes apache/incubator-metron#403
>- METRON-637: Add a STATS_BIN function to Stellar. closes
>apache/incubator-metron#401
>- METRON-622 Create a Metron Docker Compose application  (merrimanr)
>closes apache/incubator-metron#393
>- METRON-643: Stellar function documentation needs to be updated
>closes apache/incubator-metron#407
>- METRON-645 Unable to Start Fastcapa Test Environment (nickwallen)
>closes apache/incubator-metron#410
>- METRON-647 Parser unit test failures due to assumed year values
>(justinleet) closes apache/incubator-metron#412
>- METRON-639: The Network Stellar functions need to have better unit

Re: question on 'abandoned' pr

[P. Taylor Goetz]

> On Jan 26, 2017, at 9:19 PM, Otto Fowler  wrote:
> 
> PR: https://github.com/apache/incubator-metron/pull/361
> 
> This is a valid PR for building on macs, but the submitter has not
> responded to comments about changing PR title with the jira name.  I would
> very much like to land it, but I’m not sure how we want to proceed.
> 
> Originally it did not have a jira, so I created one.
> 
> How should this be handled?
> 
> Should I create my own pr based on this one and to resolve the issue?

You are fine to do that IMO. The fact that the individual opened a pull request 
shows intention to contribute, so you can safely include that commit. However,  
I would advise against squashing it. It is important to retain authorship 
history.

-Taylor

[GitHub] incubator-metron issue #425: METRON 609 Enhance Mpack to handle single-node ...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/425
  
@dlyle65535 , I see that changes necessary to single-node, in slave.py and 
elastic_slave.py, were in my METRON-634 commit rather than the METRON-609 
commit.  Sorry for the oversight.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Next Release (0.3.1) Content

[Casey Stella]

I took the liberty of adding the pull requests since the new year into the
in-progress list in the previous email.  If you own one of these and do not
believe that you can complete the review in the next week or so, let me
know and we can remove.  The only one of these that I see as mandatory is
METRON-650 because the reliance on the opensoc github repo might cause
issues with the release being acceptable (see discussion by Mike Miklavcic
with the mentors a couple weeks ago).

On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella  wrote:

> Hello Everyone!
>
> It's been almost 2 months since the last major release and I think it
> might be time to do a minor release of 0.3.1. The purpose of this email is
> multifold:
>
>- to take stock of what we have already committed
>- figure out what (if anything) is missing to make a release that
>we're proud of
>- find volunteers for the missing items
>
> For those who get email alerts on the JIRA changes, it should be no
> surprise that I have gone through and put the committed items in the 0.3.1
> release bucket.  So, let's take stock of what we have.
>
> *What's made it so far into the next release*
>
>
>
>- METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet)
>closes apache/incubator-metron#421
>- METRON-668: Remove the "tickUpdate" profile config and make the
>"init" phase not reset variables closes apache/incubator-metron#420
>- METRON-672: SolrIndexingIntegrationTest fails intermittently closes
>apache/incubator-metron#424
>- METRON-664: Make the index configuration per-writer with
>enabled/disabled closes apache/incubator-metron#419
>- METRON-600: Fix Metron Website (iraghumitra via cestella) closes
>apache/incubator-metron#399
>- METRON-666 Fix javadoc doclint errors closes
>apache/incubator-metron#418
>- METRON-659 Emulate Sensors in Development Environments (nickwallen)
>closes apache/incubator-metron#417
>- METRON-652: Extract indexing config from enrichment config closes
>apache/incubator-metron#415
>- METRON-654 Create RPM Installer for Profiler (nickwallen) closes
>apache/incubator-metron#413
>- METRON-656: Make Stellar 'in' closer to functioning like python
>closes apache/incubator-metron#416
>- METRON-532 Define Profile Period When Calling PROFILE_GET closes
>apache/incubator-metron#414
>- METRON-624: Updated Comparison/Equality Evaluations in Stellar
>closes apache/incubator-metron#404
>- METRON-644 RPM builds only work with Docker for Mac (kylerichardson)
>closes apache/incubator-metron#409
>- METRON-640: Add a Stellar function to compute shannon entropy for
>strings closes apache/incubator-metron#403
>- METRON-637: Add a STATS_BIN function to Stellar. closes
>apache/incubator-metron#401
>- METRON-622 Create a Metron Docker Compose application  (merrimanr)
>closes apache/incubator-metron#393
>- METRON-643: Stellar function documentation needs to be updated
>closes apache/incubator-metron#407
>- METRON-645 Unable to Start Fastcapa Test Environment (nickwallen)
>closes apache/incubator-metron#410
>- METRON-647 Parser unit test failures due to assumed year values
>(justinleet) closes apache/incubator-metron#412
>- METRON-639: The Network Stellar functions need to have better unit
>testing closes apache/incubator-metron#402
>- METRON-631 Broken link on fastcapa README (JonZeolla via nickwallen)
>closes apache/incubator-metron#398
>- METRON-625: Parser Filters cannot be specified from the sensor
>config closes apache/incubator-metron#396
>- METRON-587 Integration tests should use common processor
>implementations where possible (ottobackwards) closes
>apache/incubator-metron#374
>- METRON-616: Added support for float and long literals in Stellar
>closes apache/incubator-metron#392
>- METRON-580: Remove hard-coded Metron version from Ambari MPack code
>(mmiklavc) closes apache/incubator-metron#364
>- METRON-618 Eliminate Javac Warnings in metron-analytics (justinleet)
>closes apache/incubator-metron#391
>- METRON-364: Preserve Type for Arithmetic Expressions in Stellar
>closes apache/incubator-metron#390
>- METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
>apache/incubator-metron#369
>- METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT functions
>(ottobackwards) closes apache/incubator-metron#370
>- METRON-612 Clean up Error Prone generated warnings (justinleet)
>closes apache/incubator-metron#389
>- METRON-610: OnlineStatisticsProvider serialization is broken at
>random in the REPL closes apache/incubator-metron#388
>- METRON-596 Eliminate Maven warnings from build (justinleet) closes
>apache/incubator-metron#378
>- METRON-604 Mpack installs do not work on clean machines due to
>missing Elastic Curator repo (justinleet) closes 
> 

[DISCUSS] Next Release (0.3.1) Content

[Casey Stella]

Hello Everyone!

It's been almost 2 months since the last major release and I think it might
be time to do a minor release of 0.3.1. The purpose of this email is
multifold:

   - to take stock of what we have already committed
   - figure out what (if anything) is missing to make a release that we're
   proud of
   - find volunteers for the missing items

For those who get email alerts on the JIRA changes, it should be no
surprise that I have gone through and put the committed items in the 0.3.1
release bucket.  So, let's take stock of what we have.

*What's made it so far into the next release*



   - METRON-283 Migrate Geo Enrichment outside of MySQL (justinleet) closes
   apache/incubator-metron#421
   - METRON-668: Remove the "tickUpdate" profile config and make the "init"
   phase not reset variables closes apache/incubator-metron#420
   - METRON-672: SolrIndexingIntegrationTest fails intermittently closes
   apache/incubator-metron#424
   - METRON-664: Make the index configuration per-writer with
   enabled/disabled closes apache/incubator-metron#419
   - METRON-600: Fix Metron Website (iraghumitra via cestella) closes
   apache/incubator-metron#399
   - METRON-666 Fix javadoc doclint errors closes
   apache/incubator-metron#418
   - METRON-659 Emulate Sensors in Development Environments (nickwallen)
   closes apache/incubator-metron#417
   - METRON-652: Extract indexing config from enrichment config closes
   apache/incubator-metron#415
   - METRON-654 Create RPM Installer for Profiler (nickwallen) closes
   apache/incubator-metron#413
   - METRON-656: Make Stellar 'in' closer to functioning like python closes
   apache/incubator-metron#416
   - METRON-532 Define Profile Period When Calling PROFILE_GET closes
   apache/incubator-metron#414
   - METRON-624: Updated Comparison/Equality Evaluations in Stellar closes
   apache/incubator-metron#404
   - METRON-644 RPM builds only work with Docker for Mac (kylerichardson)
   closes apache/incubator-metron#409
   - METRON-640: Add a Stellar function to compute shannon entropy for
   strings closes apache/incubator-metron#403
   - METRON-637: Add a STATS_BIN function to Stellar. closes
   apache/incubator-metron#401
   - METRON-622 Create a Metron Docker Compose application  (merrimanr)
   closes apache/incubator-metron#393
   - METRON-643: Stellar function documentation needs to be updated closes
   apache/incubator-metron#407
   - METRON-645 Unable to Start Fastcapa Test Environment (nickwallen)
   closes apache/incubator-metron#410
   - METRON-647 Parser unit test failures due to assumed year values
   (justinleet) closes apache/incubator-metron#412
   - METRON-639: The Network Stellar functions need to have better unit
   testing closes apache/incubator-metron#402
   - METRON-631 Broken link on fastcapa README (JonZeolla via nickwallen)
   closes apache/incubator-metron#398
   - METRON-625: Parser Filters cannot be specified from the sensor config
   closes apache/incubator-metron#396
   - METRON-587 Integration tests should use common processor
   implementations where possible (ottobackwards) closes
   apache/incubator-metron#374
   - METRON-616: Added support for float and long literals in Stellar
   closes apache/incubator-metron#392
   - METRON-580: Remove hard-coded Metron version from Ambari MPack code
   (mmiklavc) closes apache/incubator-metron#364
   - METRON-618 Eliminate Javac Warnings in metron-analytics (justinleet)
   closes apache/incubator-metron#391
   - METRON-364: Preserve Type for Arithmetic Expressions in Stellar closes
   apache/incubator-metron#390
   - METRON-585 Pcap Replay Fails to Stop (nickwallen) closes
   apache/incubator-metron#369
   - METRON-586 STELLAR should have FILL_LEFT and FILL_RIGHT functions
   (ottobackwards) closes apache/incubator-metron#370
   - METRON-612 Clean up Error Prone generated warnings (justinleet) closes
   apache/incubator-metron#389
   - METRON-610: OnlineStatisticsProvider serialization is broken at random
   in the REPL closes apache/incubator-metron#388
   - METRON-596 Eliminate Maven warnings from build (justinleet) closes
   apache/incubator-metron#378
   - METRON-604 Mpack installs do not work on clean machines due to missing
   Elastic Curator repo (justinleet) closes apache/incubator-metron#385
   - METRON-607: Enrichment doc improvement and test cleanup (mmiklavc)
   closes apache/incubator-metron#386
   - METRON-606 Profiler Overwriting Previously Written Values (nickwallen)
   closes apache/incubator-metron#387
   - METRON-591: Make the website in compliance with ASF standards closes
   apache/incubator-metron#373
   - METRON-597 Sporadic Failures of Profiler Integration Tests
   (nickwallen) closes apache/incubator-metron#383
   - METRON-595 Elasticsearch Writer only uses One IP Address
   (JonathanRider via dlyle65535) closes apache/incubator-metron#379
   - METRON-593 Enable an automated static analysis tool in the build
   (justinleet) closes apache/incubator-metron#376
   - 

[GitHub] incubator-metron issue #426: METRON-675: Make Threat Triage rules able to be...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/426
  
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).

## Preliminaries

It is helpful to install the elasticsearch head plugin:
* `/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head`

Also, set an environment variable to indicate `METRON_HOME`:
* `export METRON_HOME=/usr/metron/0.3.0` 

## Adjust configs to Bro
We will adjust the bro topology to have a couple of threat triage rules:
* Edit `$METRON_HOME/config/zookeeper/enrichment/bro.json` as follows:
```
{
  "enrichment" : {
"fieldMap": {
  "geo": ["ip_dst_addr", "ip_src_addr"],
  "host": ["host"]
}
  },
  "threatIntel": {
"fieldMap": {
  "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"],
  "stellar" : {
  "config" : {
 "is_alert" : "ip_dst_port == 80 or ip_dst_port == 5353"
 }
  }
},
"fieldToTypeMap": {
  "ip_src_addr" : ["malicious_ip"],
  "ip_dst_addr" : ["malicious_ip"]
},
"triageConfig" : {
   "riskLevelRules" : [
  {
 "name" : "web",
 "comment" : "Bump risk if web connection",
 "rule" : "ip_dst_port == 80",
 "score" : 10
  },
  {
 "name" : "dns",
 "comment" : "Bump risk if dns connection",
 "rule" : "ip_dst_port == 5353",
 "score" : 20
  }
  ],
   "aggregator" : "MAX"
 }
  }
}
```

This should create 2 rules:
* set the triage level to 20 if the destination port is a DNS port
* set the triage level to 10 if the destination port is a web port

Ensure via the elasticsearch head plugin that the following is true:
* All bro messages with `is_alert == true` and `ip_dst_port == 5353` have a 
`threat:triage:level` of 20
* All bro messages with `is_alert == true` and `threat:triage:level == 20` 
have a `ip_dst_port` of 20
* All bro messages with `is_alert == true` and `ip_dst_port == 80` have a 
`threat:triage:level` of 10
* All bro messages with `is_alert == true` and `threat:triage:level == 10` 
have a `ip_dst_port` of 10

### Test Case: Stellar Management Functions
* Upload the management functions via `scp 
metron-platform/metron-management/target/metron-management-0.3.0.jar 
root@node1:/usr/metron/0.3.0/lib`
* Create a file with the following contents named `~/script.stellar`
```
# First we get the squid enrichment config from zookeeper.
# If it is not there, which it is not by default, a suitable default
# config will be specified.
squid_enrichment_config := CONFIG_GET('ENRICHMENT', 'squid')
# We should not have any threat triage rules
THREAT_TRIAGE_PRINT(squid_enrichment_config)
# Just for illustration, we can create a threat alert if the country of the 
domain registered
# is non-US, then we can make an alert.  To do that, we need to create an 
is_alert field on the message.
#
# I know that maps get folded into the message, so that whois_info 
enrichment is going to create a few fields:
#  * domain mapped to whois_info.domain
#  * registrar mapped to whois_info.registrar
#  * home_country mapped to whois_info.home_country
#  * owner mapped to whois_info.owner
whois_info.home_country := 'US'
# Now with this, we can create a rule or two to triage these alerts.
# This means associating a rule as described by a stellar expression that 
returns true or false with a score
# Also associated with this ruleset is an aggregation function, the default 
of which is MAX.
# Now we can make a couple rules:
#  * If the message is an alert and from a non-us whois source, we can set 
the level to 10
#  * If the message is an alert and non-local, we can set the level to 20
#  * If the message is an alert and both non-local and non-us, then we can 
set the level to 50
# If multiple rules hit, then we should take the max (default behavior)
non_us := whois_info.home_country != 'US'
is_local := IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, 
'192.168.0.0/21')
is_both := whois_info.home_country != 'US' && IN_SUBNET( if 
IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
rules := [ { 'name' : 'is non-us', 'rule' : SHELL_GET_EXPRESSION('non_us'), 
'score' : 10 } , { 'name' : 'is local', 'rule' : 
SHELL_GET_EXPRESSION('is_local'), 'score' : 20 } , { 'name' : 'both non-us and 
local', 'comment' : 'union of both rules.',  'rule' : 

[GitHub] incubator-metron pull request #427: METRON-676 Create Zeppelin Notebook for ...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/incubator-metron/pull/427

METRON-676 Create Zeppelin Notebook for YAF Telemetry

Created a Zeppelin Notebook that serves as a basic template for a 
situational awareness dashboard for the YAF flow telemetry produced by Metron. 

In practice, this notebook should be enhanced and customized to leverage 
enrichments specific to your production environment.  The notebook provides a 
fair introduction into the mechanics of using Zeppelin/Spark to work with the 
telemetry that is archived by Metron in HDFS.  

The Zeppelin Notebook is deployed with Metron through the MPack and can be 
installed by using the "Metron" > "Service Actions" > "Zeppelin Notebook 
Import" action in Ambari.

[METRON-676](https://issues.apache.org/jira/browse/METRON-676) contains a 
screen capture of the dashboard when it is run with roughly 7 days of archived 
telemetry data.

### Dependency

This change is dependent on #423 , which is why you will see those commits 
included here.  Once #423 hits master, I will rebase on master.

### Testing

I tested this change by following these steps.

 Build It

* Build Metron
```
cd incubator-metron
mvn clean package -DskipTests
```
* Start Docker on your build machine
* Build Metron RPMs
```
cd metron-deployment
mvn clean package -Pbuild-rpms -DskipTests
```
* Build Ambari MPack
```
cd metron-deployment
mvn clean package
```

 Setup Test VM
* Launch Vagrant and install Ambari only
```
cd metron-deployment/vagrant/quick-dev-platform
vagrant --ansible-tags=ambari up
```
* Copy artifacts to VM
```
scp metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm 
vagrant@node1:/tmp
scp 
metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz
 vagrant@node1:/tmp
```
* Stage RPMS
```
vagrant ssh
sudo su -
mkdir /localrepo
cp /tmp/metron*.rpm /localrepo
```

 Install Ambari MPack
* Install MPack
```
ambari-server install-mpack 
--mpack=/tmp/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
```
* Restart Ambari
```
service ambari-server restart
```

 Deploy Metron with MPack
* You may need to clear the browser cache to see the additional options 
installed by the MPack.
* Login to Ambari at http://node1:8080
* Click "Actions" -> "Add Services", then choose "Metron"

 Install Zeppelin Notebooks
* Login to Ambari at http://node1:8080.
* In Ambari, click "Metron" > "Service Actions" > "Zeppelin Notebook Import"
* Wait for the action to complete in Ambari.
* Login to Zeppelin at http://node1:9995
* Search for the notebook named "Metron - YAF Telemetry"  



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-676

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/427.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #427






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #425: METRON 609 Enhance Mpack to handle single-node ...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/425
  
@dlyle65535 , makes sense to me.  METRON-671 is very important for 
rationalizing our install scenarios, and clearly these fixes can be pipelined 
as you describe.  I'm super glad you're picking these up.

FYI, the METRON-634 fixes included here have all been proven out in my 
single-node installer, which I've used quite a bit over the past month. Also, 
all but the last 2 of the 12 items "not affecting the Ambari database" are bug 
fixes, not enhancements, without which some piece or other of the ES 
installation doesn't work correctly in today's Mpack, at least with Centos7. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #423: METRON-270: Add Zeppelin to the platform

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/423
  
@nickwallen , glad you got it working.  Just BTW, the [wiki 
page](https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install)
 recently posted by Dima Kovalyov (sorry, can't find the github uid) has fairly 
detailed step-by-step instructions for Ambari install, which it looks like you 
pretty well replicated.  It will need updating after @dlyle65535 commits the 
stuff from METRON-609 and METRON-634 (see PR#425).

Would be great if you or @justinleet add the Zeppelin steps to that wiki 
page :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/423#discussion_r98066760
  
--- Diff: metron-deployment/README.md ---
@@ -133,6 +133,8 @@ Notably, the URL for the GeoIP database that is 
preloaded (and is prefilled by d
 
 After installation, a custom action is available in Ambari (where stop / 
start services are) to install Elasticsearch templates.  Similar to this, a 
custom Kibana action to Load Template is available.
 
+Another custom action is available in Ambari to import Zeppelin dashboards.
--- End diff --

Should we link from this mention here to the more detailed information that 
you have in `metron-platform/metron-indexing/README.md`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #397: METRON-627: Add HyperLogLogPlus implementation ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/397
  
@mmiklavc Can you please merge master in and commit this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #423: METRON-270: Add Zeppelin to the platform

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/423
  
I had the wrong directory.  If I put a Notebook in 
`/usr/metron/0.3.0/config/zeppelin/` then it works properly.  Nice!

```
2017-01-26 18:12:53,238 - The hadoop conf dir 
/usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for 
version 2.5.0.0-1245
2017-01-26 18:12:53,238 - Checking if need to create versioned conf dir 
/etc/hadoop/2.5.0.0-1245/0
2017-01-26 18:12:53,240 - call[('ambari-python-wrap', 
'/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', 
'--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 
'sudo': True, 'quiet': False, 'stderr': -1}
2017-01-26 18:12:53,273 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 
exist already', '')
2017-01-26 18:12:53,273 - checked_call[('ambari-python-wrap', 
'/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', 
'--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 
'sudo': True, 'quiet': False}
2017-01-26 18:12:53,295 - checked_call returned (0, '')
2017-01-26 18:12:53,296 - Ensuring that hadoop has the correct symlink 
structure
2017-01-26 18:12:53,296 - Using hadoop conf dir: 
/usr/hdp/current/hadoop-client/conf
2017-01-26 18:12:53,299 - Execute['curl -s -XPOST 
http://node1:9995/api/notebook/import -d 
"@/usr/metron/0.3.0/config/zeppelin/Metron - YAF Telemetry.json"'] 
{'logoutput': True}
{"status":"CREATED","message":"","body":"2C88YEF19"}

Command completed successfully!
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #425: METRON 609 Enhance Mpack to handle single-node ...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/425
  
Thanks, @dlyle65535 .  Couple more comments:

- Also included is the small change proposed in METRON-641 to use {0} 
instead of {} in python format strings in kibana_master.py.  It shouldn't be 
necessary for Python 2.7, but these are the only usages of the 2.7 behavior, 
and there's no harm in making it also work with Python 2.6.

- The biggest deficiency, as far as I'm concerned, is the need to use a 
different, sparser template for a single-node "elasticsearch.master.yaml.j2".  
Presumably only one or two of the removed parameters are actually a problem, 
and presumably even those don't really have to be removed but simply need a 
different value.  But it looked like a big job to figure out.  Perhaps someone 
more familiar with Elasticsearch would be able to solve it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Cannot close JIRAs I didn't originally request

[Billie Rinaldi]

It appears that Contributors are missing from the Transition Issues
permission, which seems odd since Contributors do have the Close Issues and
Resolve Issues permissions. This may be keeping Jon from transitioning the
issue to Done, since Metron has a nonstandard workflow which has Done
instead of Resolved as the completed state for issues. The workflow may
also be why there is no way to specify the reason for the resolution of the
ticket. If we want to make any adjustments to the permissions or workflow,
that will require an INFRA ticket.

On Thu, Jan 26, 2017 at 8:30 AM, zeo...@gmail.com  wrote:

> I assigned a JIRA (METRON-354
> ) to me that was
> reported
> by someone else, and it appears that I don't have the permissions to close
> it.  For clarity, I didn't have access to close it before I assigned it to
> myself either.  Would someone be willing to delegate me the appropriate
> access?  Alternatively, if you're not comfortable providing that level of
> access, or if it is not simple to do, I would be happy to ask that the
> original requester close their own ticket.
>
> Thanks,
>
> Jon
> --
>
> Jon
>
> Sent from my mobile device
>

Re: Cannot close JIRAs I didn't originally request

[Anand Subramanian]

Hello Jon,

I went ahead and marked the defect as 'Done'. 

However, I could not find a way to specify the reason for resolution (not 
reproducible, for e.g.) when closing the ticket. I am not sure if this is by 
design, or that additional privileges are required for more fields to show up.

Regards,
Anand




On 1/26/17, 10:00 PM, "zeo...@gmail.com"  wrote:

>I assigned a JIRA (METRON-354
>) to me that was reported
>by someone else, and it appears that I don't have the permissions to close
>it.  For clarity, I didn't have access to close it before I assigned it to
>myself either.  Would someone be willing to delegate me the appropriate
>access?  Alternatively, if you're not comfortable providing that level of
>access, or if it is not simple to do, I would be happy to ask that the
>original requester close their own ticket.
>
>Thanks,
>
>Jon
>-- 
>
>Jon
>
>Sent from my mobile device

[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/423#discussion_r98044156
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
 ---
@@ -187,6 +187,14 @@
   600
 
   
+  
+ZEPPELIN_DASHBOARD_INSTALL
--- End diff --

Perfect.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/423#discussion_r98042802
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
 ---
@@ -187,6 +187,14 @@
   600
 
   
+  
+ZEPPELIN_DASHBOARD_INSTALL
--- End diff --

Seems like they use "Import" instead of "Install".  How about "Zeppelin 
Notebook Import"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/423#discussion_r98042135
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
 ---
@@ -115,6 +116,16 @@ def elasticsearch_template_delete(self, env):
 yaf_cmd = ambari_format('curl -s -XDELETE 
"http://{es_http_url}/yaf_index*;')
 Execute(yaf_cmd, logoutput=True)
 
+def zeppelin_dashboard_install(self, env):
+from params import params
+env.set_params(params)
+
+for dirName, subdirList, files in 
os.walk(params.metron_config_zeppelin_path):
--- End diff --

I'll add logging for that.  You're right, in the case that there are no 
notebooks, it'd be helpful to see what it's using.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #423: METRON-270: Add Zeppelin to the platform

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/423#discussion_r98040142
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
 ---
@@ -115,6 +116,16 @@ def elasticsearch_template_delete(self, env):
 yaf_cmd = ambari_format('curl -s -XDELETE 
"http://{es_http_url}/yaf_index*;')
 Execute(yaf_cmd, logoutput=True)
 
+def zeppelin_dashboard_install(self, env):
+from params import params
+env.set_params(params)
+
+for dirName, subdirList, files in 
os.walk(params.metron_config_zeppelin_path):
--- End diff --

Would it make sense to log the path that it is looking for the Notebooks 
in; `params.metron_config_zeppelin_path`?  I'd like to see this in the Ambari 
status output, especially if it doesn't find any Notebooks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #423: METRON-270: Add Zeppelin to the platform

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/423
  
So far, I have not been able to get the "Zeppelin Dashboard Install" 
mechanism to work. 
 
I am not aware of step-by-step instructions to use the MPack, but maybe I 
am missing them.  Thought it would be beneficial to document here since it took 
me a while to piece all this together.  Here is exactly what I did.  What did I 
do wrong?


### Build

* Build Metron
```
cd incubator-metron
mvn clean package -DskipTests
```

* Start Docker on your build machine

* Build Metron RPMs
```
mvn clean package -Dbuild-rpms -DskipTests
```

* Validate Metron RPMs
```
$ find ./ -name "*.rpm"

.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-common-0.3.0-201701261504.noarch.rpm

.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-data-management-0.3.0-201701261504.noarch.rpm

.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-elasticsearch-0.3.0-201701261504.noarch.rpm

.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-enrichment-0.3.0-201701261504.noarch.rpm

.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-indexing-0.3.0-201701261504.noarch.rpm

.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-parsers-0.3.0-201701261504.noarch.rpm

.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-pcap-0.3.0-201701261504.noarch.rpm

.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-profiler-0.3.0-201701261504.noarch.rpm

.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-solr-0.3.0-201701261504.noarch.rpm

.//metron-deployment/packaging/docker/rpm-docker/SRPMS/metron-0.3.0-201701261504.src.rpm
```

* Build Ambari MPack
```
cd metron-deployment
mvn clean package
```

* Validate Ambari MPack
```
$ find . -name "*mpack*.tar.gz"
./packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz
```

### Create Test VM

* Launch Vagrant and install Ambari only
```
cd metron-deployment/vagrant/quick-dev
vagrant --ansible-tags=ambari up
```

* Copy Artifacts to VM
```
scp 
incubator-metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm
 vagrant@node1:/tmp
scp 
incubator-metron/metron-deploment/packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz
 vagrant@node1:/tmp
```

* Prepare RPMS
```
vagrant ssh
sudo su -
mkdir /localrepo
cp /tmp/metron*.rpm /localrepo
```

### Install Ambari MPack

* Install MPack
```
ambari-server install-mpack 
--mpack=/tmp/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
service ambari-server restart
```

### Deploy Metron with MPack

* You may need to clear the browser cache to see the additional options 
installed by the MPack.

* Login to Ambari at http://node1:8080

* Click Add Services

* Select "Metron" and then continue the install

* SUCCESS!  Validated that I was prompted to install Zeppelin and Spark as 
dependencies

### Install Zeppelin Notebooks

* Create a Zeppelin notebook and drop it at `/usr/metron/0.3.0/zeppelin/`.

* In Ambari, Click Metron > Service Actions > Zeppelin Dashboard Install.  

* This is what I see from the Ambari status window.
```
2017-01-26 16:45:29,926 - The hadoop conf dir 
/usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for 
version 2.5.0.0-1245
2017-01-26 16:45:29,926 - Checking if need to create versioned conf dir 
/etc/hadoop/2.5.0.0-1245/0
2017-01-26 16:45:29,927 - call[('ambari-python-wrap', 
'/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', 
'--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 
'sudo': True, 'quiet': False, 'stderr': -1}
2017-01-26 16:45:29,947 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 
exist already', '')
2017-01-26 16:45:29,948 - checked_call[('ambari-python-wrap', 
'/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', 
'--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 
'sudo': True, 'quiet': False}
2017-01-26 16:45:29,968 - checked_call returned (0, '')
2017-01-26 16:45:29,972 - Ensuring that hadoop has the correct symlink 
structure
2017-01-26 16:45:29,972 - Using hadoop conf dir: 
/usr/hdp/current/hadoop-client/conf

Command completed successfully!
```

* FAILURE! I then go to Zeppelin at `http://node1:9995/` and search for the 
notebook.  The Notebook does not seem to be installed.





---
If your project is set up for it, you can reply to this email and have your
reply appear on 

[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/421


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #421: METRON-283 Migrate Geo Enrichment outside of My...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/421
  
reaffirmed +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/405
  
Here's a fun fact- quick-dev uses 2.6.6. Had to make this exact change 
while unifying Ansible with the MPack, so if nothing else, we'll get it there. 
Still I'd like @DimDroll to get the deserved credit, so I'd be happy for this 
one to get there first.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #421: METRON-283 Migrate Geo Enrichment outside of My...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/421
  
+1 looks good


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #426: METRON-675: Make Threat Triage rules abl...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/426

METRON-675: Make Threat Triage rules able to be assigned names and comments

There may be many, many threat triage rules. To help organize these, we 
should make them slightly more complex than a simple key/value as we have it 
now. We should add optional name and optional comment fields.

This essentially makes the risk level rules slightly more complex.  The 
format goes from:
```
"riskLevelRules" : {
  "stellar expression" : numeric score
}
```
to:
```
"riskLevelRules" : [
  {
 "name" : "optional name",
 "comment" : "optional comment",
 "rule" : "stellar expression",
 "score" : numeric score
  }
]
```
This is NOT backwards compatible, but I think it's more explicit and a bit 
more clear.

Testing plan to come in a follow-on comment.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-675

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/426.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #426


commit 2d9c129e2be95d635d5c014415087b7a13a678db
Author: cstella 
Date:   2017-01-26T16:15:01Z

METRON-675: Add name and description to threat triage rules.

commit 8639d9967afb2add2035aa57fa60d4cc17cbb117
Author: cstella 
Date:   2017-01-26T16:21:34Z

forgot license




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #421: METRON-283 Migrate Geo Enrichment outside of My...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/421
  
Still +1 @justinleet - thanks for checking.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Cannot close JIRAs I didn't originally request

[zeo...@gmail.com]

I assigned a JIRA (METRON-354
) to me that was reported
by someone else, and it appears that I don't have the permissions to close
it.  For clarity, I didn't have access to close it before I assigned it to
myself either.  Would someone be willing to delegate me the appropriate
access?  Alternatively, if you're not comfortable providing that level of
access, or if it is not simple to do, I would be happy to ask that the
original requester close their own ticket.

Thanks,

Jon
-- 

Jon

Sent from my mobile device

Re: Reporting Issues Wiki

[zeo...@gmail.com]

Okay, I've modified the wiki page
 to
point to dev@.  Thanks!

Jon

On Thu, Jan 26, 2017 at 11:13 AM Billie Rinaldi  wrote:

> It's okay, we don't have to vote to fix a typo. It should be user@ or dev@
> instead of issues@. David, we have the ability to administer a few
> permissions for the Metron JIRA, so people should email us first before
> trying INFRA.
>
> On Thu, Jan 26, 2017 at 7:43 AM, zeo...@gmail.com 
> wrote:
>
> > Okay.  Since that wiki page has been voted on in the past, I'm hesitant
> to
> > modify it.  I will defer that modification and/or voting procedure to
> > someone on the PMC.
> >
> > Jon
> >
> > On Thu, Jan 26, 2017 at 10:38 AM David Lyle 
> wrote:
> >
> > > No, that's gotta be an error on the wiki. For wiki issues, I'd use
> user@
> > ,
> > > dev@ or just write a Jira. For issues with Jira itself, I'd contact
> > infra,
> > > see https://www.apache.org/dev/infrastructure.html.
> > >
> > > Thanks!
> > >
> > > -D...
> > >
> > >
> > > On Thu, Jan 26, 2017 at 1:32 AM, Matt Foley  wrote:
> > >
> > > > Actually, it is a little odd that we would ask users to email to
> > issues@
> > > .
> > > > For most projects that is a broad distro list, where Apache Jira
> sends
> > > > automatic notifications of every change to every ticket in the
> > project’s
> > > > (METRON) Jira project.  It would normally be expected to be
> essentially
> > > > read-only, altho evidently ezmlm is set up to allow apache.org email
> > > > addresses (meaning you’re a committer or PMC member) to send to it.
> > > Being
> > > > a subscriber/recipient of it is unrelated to the privs for sending to
> > it
> > > > (unlike users@ or dev@).
> > > >
> > > > James or David, has issues@ for the metron project been set up
> > > > differently than the Apache norm, so it is in fact an appropriate
> place
> > > for
> > > > non-committer users to report problems with Jira?
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 1/25/17, 7:24 PM, "zeo...@gmail.com"  wrote:
> > > >
> > > > On the Reporting Issues
> > > >  > Reporting+Issues
> > > >
> > > > wiki
> > > > page it mentions to "Please report issues related to the
> JIRA/Wiki
> > to
> > > > iss...@metron.incubator.apache.org".  I have a personal
> permission
> > > > issue
> > > > with JIRA that I was attempting to email to that list and I keep
> > > > getting
> > > > the following error message:
> > > >
> > > >
> > > > Hi. This is the qmail-send program at apache.org.
> > > > I'm afraid I wasn't able to deliver your message to the following
> > > > addresses.
> > > > This is a permanent error; I've given up. Sorry it didn't work
> out.
> > > >
> > > > :
> > > > Must be sent from an @apache.org address or an address in LDAP.
> > > >
> > > > --- Below this line is a copy of the message.
> > > > ...
> > > >
> > > >
> > > > Note that I am sending the email to "issues@metron.incubator.
> > > > apache.org",
> > > > however it is replying with the text "iss...@metron.apache.org".
> > I
> > > > have
> > > > also made sure to join the mailing list, as shown below:
> > > >
> > > >
> > > > Hi! This is the ezmlm program. I'm managing the
> > > > iss...@metron.incubator.apache.org mailing list.
> > > >
> > > > Acknowledgment: I have added the address
> > > >
> > > > zeo...@gmail.com
> > > >
> > > > to the issues mailing list.
> > > >
> > > > Welcome to iss...@metron.incubator.apache.org!
> > > >
> > > >
> > > > Am I missing something here?  The actual issue I'm having is not
> > that
> > > > urgent, I think making sure the wiki has the appropriate
> > instructions
> > > > is
> > > > more important.
> > > >
> > > > Jon
> > > > --
> > > >
> > > > Jon
> > > >
> > > > Sent from my mobile device
> > > >
> > > >
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
> > Sent from my mobile device
> >
>
-- 

Jon

Sent from my mobile device

Re: Reporting Issues Wiki

[Billie Rinaldi]

It's okay, we don't have to vote to fix a typo. It should be user@ or dev@
instead of issues@. David, we have the ability to administer a few
permissions for the Metron JIRA, so people should email us first before
trying INFRA.

On Thu, Jan 26, 2017 at 7:43 AM, zeo...@gmail.com  wrote:

> Okay.  Since that wiki page has been voted on in the past, I'm hesitant to
> modify it.  I will defer that modification and/or voting procedure to
> someone on the PMC.
>
> Jon
>
> On Thu, Jan 26, 2017 at 10:38 AM David Lyle  wrote:
>
> > No, that's gotta be an error on the wiki. For wiki issues, I'd use user@
> ,
> > dev@ or just write a Jira. For issues with Jira itself, I'd contact
> infra,
> > see https://www.apache.org/dev/infrastructure.html.
> >
> > Thanks!
> >
> > -D...
> >
> >
> > On Thu, Jan 26, 2017 at 1:32 AM, Matt Foley  wrote:
> >
> > > Actually, it is a little odd that we would ask users to email to
> issues@
> > .
> > > For most projects that is a broad distro list, where Apache Jira sends
> > > automatic notifications of every change to every ticket in the
> project’s
> > > (METRON) Jira project.  It would normally be expected to be essentially
> > > read-only, altho evidently ezmlm is set up to allow apache.org email
> > > addresses (meaning you’re a committer or PMC member) to send to it.
> > Being
> > > a subscriber/recipient of it is unrelated to the privs for sending to
> it
> > > (unlike users@ or dev@).
> > >
> > > James or David, has issues@ for the metron project been set up
> > > differently than the Apache norm, so it is in fact an appropriate place
> > for
> > > non-committer users to report problems with Jira?
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 1/25/17, 7:24 PM, "zeo...@gmail.com"  wrote:
> > >
> > > On the Reporting Issues
> > >  Reporting+Issues
> > >
> > > wiki
> > > page it mentions to "Please report issues related to the JIRA/Wiki
> to
> > > iss...@metron.incubator.apache.org".  I have a personal permission
> > > issue
> > > with JIRA that I was attempting to email to that list and I keep
> > > getting
> > > the following error message:
> > >
> > >
> > > Hi. This is the qmail-send program at apache.org.
> > > I'm afraid I wasn't able to deliver your message to the following
> > > addresses.
> > > This is a permanent error; I've given up. Sorry it didn't work out.
> > >
> > > :
> > > Must be sent from an @apache.org address or an address in LDAP.
> > >
> > > --- Below this line is a copy of the message.
> > > ...
> > >
> > >
> > > Note that I am sending the email to "issues@metron.incubator.
> > > apache.org",
> > > however it is replying with the text "iss...@metron.apache.org".
> I
> > > have
> > > also made sure to join the mailing list, as shown below:
> > >
> > >
> > > Hi! This is the ezmlm program. I'm managing the
> > > iss...@metron.incubator.apache.org mailing list.
> > >
> > > Acknowledgment: I have added the address
> > >
> > > zeo...@gmail.com
> > >
> > > to the issues mailing list.
> > >
> > > Welcome to iss...@metron.incubator.apache.org!
> > >
> > >
> > > Am I missing something here?  The actual issue I'm having is not
> that
> > > urgent, I think making sure the wiki has the appropriate
> instructions
> > > is
> > > more important.
> > >
> > > Jon
> > > --
> > >
> > > Jon
> > >
> > > Sent from my mobile device
> > >
> > >
> > >
> > >
> >
> --
>
> Jon
>
> Sent from my mobile device
>

Re: Reporting Issues Wiki

[David Lyle]

No, that's gotta be an error on the wiki. For wiki issues, I'd use user@,
dev@ or just write a Jira. For issues with Jira itself, I'd contact infra,
see https://www.apache.org/dev/infrastructure.html.

Thanks!

-D...


On Thu, Jan 26, 2017 at 1:32 AM, Matt Foley  wrote:

> Actually, it is a little odd that we would ask users to email to issues@.
> For most projects that is a broad distro list, where Apache Jira sends
> automatic notifications of every change to every ticket in the project’s
> (METRON) Jira project.  It would normally be expected to be essentially
> read-only, altho evidently ezmlm is set up to allow apache.org email
> addresses (meaning you’re a committer or PMC member) to send to it.  Being
> a subscriber/recipient of it is unrelated to the privs for sending to it
> (unlike users@ or dev@).
>
> James or David, has issues@ for the metron project been set up
> differently than the Apache norm, so it is in fact an appropriate place for
> non-committer users to report problems with Jira?
>
> Thanks,
> --Matt
>
>
> On 1/25/17, 7:24 PM, "zeo...@gmail.com"  wrote:
>
> On the Reporting Issues
> 
> wiki
> page it mentions to "Please report issues related to the JIRA/Wiki to
> iss...@metron.incubator.apache.org".  I have a personal permission
> issue
> with JIRA that I was attempting to email to that list and I keep
> getting
> the following error message:
>
>
> Hi. This is the qmail-send program at apache.org.
> I'm afraid I wasn't able to deliver your message to the following
> addresses.
> This is a permanent error; I've given up. Sorry it didn't work out.
>
> :
> Must be sent from an @apache.org address or an address in LDAP.
>
> --- Below this line is a copy of the message.
> ...
>
>
> Note that I am sending the email to "issues@metron.incubator.
> apache.org",
> however it is replying with the text "iss...@metron.apache.org".  I
> have
> also made sure to join the mailing list, as shown below:
>
>
> Hi! This is the ezmlm program. I'm managing the
> iss...@metron.incubator.apache.org mailing list.
>
> Acknowledgment: I have added the address
>
> zeo...@gmail.com
>
> to the issues mailing list.
>
> Welcome to iss...@metron.incubator.apache.org!
>
>
> Am I missing something here?  The actual issue I'm having is not that
> urgent, I think making sure the wiki has the appropriate instructions
> is
> more important.
>
> Jon
> --
>
> Jon
>
> Sent from my mobile device
>
>
>
>

[GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/405
  
Hi, thanks for the contribution! Per Matt's comments, we do require Python 
2.7. I don't believe that's in our main README, but it is listed in our 
deployment READMEs, e.g. 
https://github.com/apache/incubator-metron/tree/master/metron-deployment/vagrant/quick-dev-platform

That being said, we have made this exact fix/change in other files in the 
Ambari installation code and it should be here as well. Also agreed with Matt's 
comments about the other 3 braces.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/405
  
Yup, please change the other 3 and we'll get this in, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Gratuating to Apache Top Level Project

[Justin Leet]

I agree, +1.

On Thu, Jan 26, 2017 at 10:20 AM, Ryan Merriman  wrote:

> I think we're ready +1
>
> On Wed, Jan 25, 2017 at 7:06 PM, Nick Allen  wrote:
>
> > +1 I think we clearly meet all of those criteria.  Glad to see the
> project
> > mature and grow.
> >
> > On Mon, Jan 23, 2017 at 7:09 PM, James Sirota 
> wrote:
> >
> > > I think the Apache Incubation was very valuable learning experience for
> > > us, but it seems like we are ready to become a top-level project.  We
> > have
> > > been in incubation since 2015-12-06 and under the guidance of our
> Mentors
> > > we had a clean build (0.3.0), we learned how to function well as a
> > > community (as defined by Apache), and if you look at our maturity level
> > > checklist we meet the criteria of the a mature Apache project. (
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=66852119)
> > >
> > >
> > > Do you think we are ready to graduate?  Should we start putting a case
> > > together for graduation?
> > >
> > > ---
> > > Thank you,
> > >
> > > James Sirota
> > > PPMC- Apache Metron (Incubating)
> > > jsirota AT apache DOT org
> > >
> >
> >
> >
> > --
> > Nick Allen 
> >
>

Re: [DISCUSS] Gratuating to Apache Top Level Project

[Ryan Merriman]

I think we're ready +1

On Wed, Jan 25, 2017 at 7:06 PM, Nick Allen  wrote:

> +1 I think we clearly meet all of those criteria.  Glad to see the project
> mature and grow.
>
> On Mon, Jan 23, 2017 at 7:09 PM, James Sirota  wrote:
>
> > I think the Apache Incubation was very valuable learning experience for
> > us, but it seems like we are ready to become a top-level project.  We
> have
> > been in incubation since 2015-12-06 and under the guidance of our Mentors
> > we had a clean build (0.3.0), we learned how to function well as a
> > community (as defined by Apache), and if you look at our maturity level
> > checklist we meet the criteria of the a mature Apache project. (
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=66852119)
> >
> >
> > Do you think we are ready to graduate?  Should we start putting a case
> > together for graduation?
> >
> > ---
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
> >
>
>
>
> --
> Nick Allen 
>

Re: [DISCUSS] Error Indexing

[Ryan Merriman]

Jon, I misread the code in the GenericEnrichmentBolt.  The error is
forwarded on so no issues there.

Defaulting to the common fields makes sense.  I will dig into the
GenericEnrichmentBolt more, maybe there is a way to get the error fields
without having to significantly change things.  Any opinion on a hashing
algorithm?

On Wed, Jan 25, 2017 at 9:37 PM, zeo...@gmail.com  wrote:

> Although hashing the whole message is better than nothing, it misses a lot
> of the benefits we could get.
>
> While I'd love to have consistency for this field across all of the
> different error.types, it appears that may not be reasonably possible
> because of the parsers.  So, how about something like hash all of the
> constant
> fields
>  metron-platform/metron-common/src/main/java/org/apache/
> metron/common/Constants.java>
> excluding
> timestamp and original_string unless it is a parser, in which case hash the
> entire message?  This gives us some measure of event uniqueness and it can
> grow as we define additional constant fields (I recall discussing with
> someone else on the list regarding expanding those standard fields to
> include things like usernames but I can't find the specific email
> exchange).
>
> Because some enrichments can be heavily relied on, I think it makes sense
> to put a message onto the error queue when it throws an exception.  Not
> only does this help troubleshoot edge cases, but it makes issues more
> obvious when assembling a new enrichment in dev/test.  I can't think of a
> scenario currently where an enrichment would only be "best effort" and that
> I wouldn't want that error indexed and retrievable.  However, this gets
> interesting when talking about the various options to solve the "Enrich
> enrichment" discussion from earlier in the month.  We can keep that part of
> this separate though, as I don't think that's being actively pursued right
> now.
>
> Jon
>
> On Wed, Jan 25, 2017 at 10:49 AM David Lyle  wrote:
>
> RE: separate JIRA for MPack/Ansible. No objection to tracking them
> separately, but for this item to be complete, you'll need both the feature
> and the ability to install it.
>
> -D...
>
>
> On Tue, Jan 24, 2017 at 5:33 PM, Ryan Merriman 
> wrote:
>
> > Assuming we're going to write all errors to a single error topic, I think
> > it makes sense to agree on an error message schema and handle errors
> across
> > the 3 different topologies in the same way with a single implementation.
> > The implementation in ParserBolt (ErrorUtils.handleError) produces the
> most
> > verbose error object so I think it's a good candidate for the single
> > implementation.  Here is the message structure it currently produces:
> >
> > {
> >   "exception": "java.lang.Exception: there was an error",
> >   "hostname": "host",
> >   "stack": "java.lang.Exception: ...",
> >   "time": 1485295416563,
> >   "message": "there was an error",
> >   "rawMessage": "raw message",
> >   "rawMessage_bytes": [],
> >   "source.type": "bro_error"
> > }
> >
> > From our discussion so far we need to add a couple fields:  an error type
> > and hash id.  Adding these to the message looks like:
> >
> > {
> >   "exception": "java.lang.Exception: there was an error",
> >   "hostname": "host",
> >   "stack": "java.lang.Exception: ...",
> >   "time": 1485295416563,
> >   "message": "there was an error",
> >   "rawMessage": "raw message",
> >   "rawMessage_bytes": [],
> >   "source.type": "bro_error",
> >   "error.type": "parser_error",
> >   "rawMessage_hash": "dde41b9920954f94066daf6291fb58a9"
> > }
> >
> > We should also consider expanding the error types I listed earlier.
> > Instead of just having "indexing_error" we could have
> > "elasticsearch_indexing_error", "hdfs_indexing_error" and so on.
> >
> > Jon, if an exception happens in an enrichment or threat intel bolt the
> > message is passed along with no error thrown (only logged).  Everywhere
> > else I'm having trouble identifying specific fields that should be
> hashed.
> > Would hashing the message in every case be acceptable?  Do you know of a
> > place where we could hash a field instead?  On the topic of exceptions in
> > enrichments, are we ok with an error only being logged and not added to
> the
> > message or emitted to the error queue?
> >
> >
> >
> > On Tue, Jan 24, 2017 at 3:10 PM, Ryan Merriman 
> > wrote:
> >
> > > That use case makes sense to me.  I don't think it will require that
> much
> > > additional effort either.
> > >
> > > On Tue, Jan 24, 2017 at 1:02 PM, zeo...@gmail.com 
> > > wrote:
> > >
> > >> Regarding error vs validation - Either way I'm not very concerned.  I
> > >> initially assumed they would be combined and agree with that approach,
> > but
> > >> splitting them out isn't a very big deal to me either.
> > >>
> > >> Re: Ryan.  Yes, exactly.  In the case of a parser issue 

[GitHub] incubator-metron issue #421: METRON-283 Migrate Geo Enrichment outside of My...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/421
  
@cestella @dlyle65535 @nickwallen I added one more change to 
`EnrichmentIntegrationTest` to not create and destroy some tmp stuff that's not 
being used in the test.  Let me know if there's any concerns.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #421: METRON-283 Migrate Geo Enrichment outside of My...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/421
  
Ditto +1. Thanks for all the hard work!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #420: METRON-668: Remove the "tickUpdate" prof...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/420


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #424: METRON-672: SolrIndexingIntegrationTest ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/424


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #424: METRON-672: SolrIndexingIntegrationTest fails i...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/424
  
Thanks again for this.  I'm +1 on it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #421: METRON-283 Migrate Geo Enrichment outside of My...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/421
  
+1 as well, great job, Justin.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #424: METRON-672: SolrIndexingIntegrationTest fails i...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/424
  
@justinleet comments addressed, let me know if there's anything else.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #424: METRON-672: SolrIndexingIntegrationTest ...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/424

METRON-672: SolrIndexingIntegrationTest fails intermittently

This failure is due to a change in default behavior when indexing was split 
off into a separate configuration file.  The default batch size was changed 
from `5` to `1` in particular.  This, by itself, is not a problem, but the 
`IndexingIntegrationTest` (base class for Solr and Elastic search integration 
tests):
* submits the configs
* starts the indexing topology
* writes the input data

The writing of the input data may happen before the topology fully loads or 
the configuration fully loads, especially if the machine running the unit tests 
is under load (like with travis).  As a result, the first record may end up 
with the default batch size (of 1) and write out immediately because the 
indexing configs haven't loaded into zookeeper just yet.  In that circumstance, 
eventually the configs load and the batch size is set to `5`.  Meanwhile we've 
written 10 records and are expecting 10 in return, but because you wrote the 
first out already and then the next 5, we have another 4 pending to be written 
by the `BulkMessageWriterBolt`.

So, the failure scenario is as follows:
* Message 1 is received and the indexing config hasn't loaded yet, so the 
batch size is 1 and it immediately gets written out
* Message 2 - 5 are each received and the indexing config has loaded, so 
the batch size is 5 and it queues up
* Message 6 is received and the batch writes out
* Messages 7 - 10 are received, but never make a full batch, so we time out 
waiting for them to write out

The fix is to ensure that we don't write out messages to kafka until the 
configs are loaded, which is what this PR does.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-672

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #424


commit 9617e099d79920b1ffd40555c036f3826ea96e6b
Author: cstella 
Date:   2017-01-26T00:01:21Z

Fixing integration test.

commit 5e836f4eaba51783c2a699e41aba6568aa6a5f99
Author: cstella 
Date:   2017-01-26T00:52:36Z

Merge branch 'master' into METRON-672

commit b2103fdb735854a375a70809e625044f0f200a29
Author: cstella 
Date:   2017-01-26T01:12:21Z

Updating to react to comments.

commit 0f6602ccf86c0cb68d851ab5337693346bb284e4
Author: cstella 
Date:   2017-01-26T01:25:09Z

fixed commit.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #424: METRON-672: SolrIndexingIntegrationTest ...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/424


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #425: METRON 609 Enhance Mpack to handle single-node ...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/425
  
Hi @mattf-horton - thanks for putting this up! It's got my "have to haves" 
plus a bunch more good stuff. I'll get working with it today. If I discover any 
small changes, I'll put a PR on your branch.

Thanks again, this is great stuff.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #408: METRON-608 Mpack to install a single-node test ...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/408
  
@dlyle65535 , please see https://github.com/apache/incubator-metron/pull/425



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #425: METRON 609 Enhance Mpack to handle singl...

[mattf-horton]

GitHub user mattf-horton opened a pull request:

https://github.com/apache/incubator-metron/pull/425

METRON 609 Enhance Mpack to handle single-node and small-cluster installs 
of Elasticsearch

This PR is not ready for prime time, but is provided for ease of access to 
work-in-progress for:
- METRON-609 Enhance Mpack to handle single-node and small-cluster installs 
of Elasticsearch, and 
- METRON-634 Mpack bug fixes and improvements (not related to singlenode 
install).  

These are presented as two separate commits, so you can look at them 
separately if you wish.

These are the included enhancements from METRON-609:
- Enable 1-, 2-, and 3-node clusters to have a working Elasticsearch 
install via the Mpack.
- Change constraints from 1+ Masters and 3+ Slaves, to 1+ and 0+.
- Allow non-dedicated master/datanodes via boolean 
"masters_also_are_datanodes".
- Allow use of alternative single-node template via 
"single_node_elasticsearch" boolean.
- Only the 1- and 4-node clusters have been tested, last month.
- Improve various mouse-over Description fields in the GUI.
- I included the attempted validation check on (storm) num_slots = 
slots_per_supervisor * num_supervisors.  This doesn't currently work due to 
pre-existing bug in other parts of validation check, so haven't been able to 
test.

These are the included enhancements and bug fixes from METRON-634:
NOT AFFECTING THE AMBARI DATABASE:
- ES pid_dir specification and usage:
- Currently pid_dir is multiply specified in elastic-env.xml and 
params.py. The config parameter should not be over-ridden in params.py.
- PID_DIR failed to be included in /etc/sysconfig/elasticsearch. It 
needs to be added to the template in elastic-sysconfig, as it must be provided 
to ES at launch-time (else the default directory will be used).
- pid_file is specified in params.py, but is not used anywhere. (The ES 
internal launcher synthesizes it from PID_DIR, and this is appropriate.)
- JAVA_HOME needs to be provided in /etc/sysconfig/elasticsearch (templated 
in elastic-sysconfig.xml). Its absence causes Centos7 systemctl to fail the ES 
launch, unless /bin/java is defined (which it isn't necessarily).
- Also in the /etc/sysconfig/elasticsearch template in 
elastic-sysconfig.xml, the value of ES_JAVA_OPTS incorrectly spans 3 lines. The 
lines must be terminated with backslashes to effectively become a single line. 
The current inclusion of newlines in the long string value is acceptable 
(although unusual) in shellscript, but not in a systemd EnvironmentFile. 
/etc/sysconfig/elasticsearch must function as both.
- Also in ES_JAVA_OPTS, the two instances of log_dir needs to be followed 
by a slash '/'
- In elastic.py, when directories are being pre-created and permissions 
set, the file $CONF_DIR/scripts should also be pre-created. I intermittently 
hit permissions issues with this directory being created later by root, and not 
properly assigned to elastic_user.
- In several places in elastic.py, "params.elastic_user" is incorrectly 
used when "params.user_group" should be used.
- Undefined "format()" method is used in elastic.py, unnecessarily in 
File(format("/etc/sysconfig/elasticsearch")...
- Undefined "format()" method is similarly used several times unnecessarily 
in elastic_master.py
- The comments and descriptions in elastic-site.xml have multiple suggested 
improvements.
- Provide Quick Links in Ambari service page for Elasticsearch to the 
self-report pages for ES health and ES node list. (very useful for debugging)

CHANGES THAT DO AFFECT THE AMBARI DATABASE:
- pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than 
elastic-env.xml, as it is a parameter that must be provided to ES at 
launch-time, but is not something there's any reason for the admin to change in 
usual circumstances.
- conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not 
in elastic-sysconfig.xml. While it too is a parameter that must be provided to 
ES at launch-time, it is typically left to the installing admin where to put 
the config files.
- The Ambari configuration parameter names in elastic-site.xml should be 
improved in several instances to make the semantics more obvious to the human 
reader (who may not be real familiar with Elasticsearch configuration). 
Mouse-over documentation will continue to provide the ES config parameter 
equivalents. In particular, suggest:
- cluster_name -> es_cluster_name  (to distinguish ES cluster from 
Stack cluster)
- zen_discovery_ping_unicast_hosts -> es_cluster_hosts
- network_host -> network_bindings  (these are in fact interface names, 
not host names)
- There are at least two places in elasticsearch.master.yaml.j2 
(zen_discovery_ping_unicast_hosts and network_host) where needed square 
brackets are either missing