[GitHub] metron pull request #960: METRON-1424: Kerberos: Solr

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/960

METRON-1424: Kerberos: Solr

## Contributor Comments
This PR adds Kerberos support for Solr in Metron.  This has been verified 
in full dev using the following steps:

1. Spin up full dev
2. Stop and remove Elasticsearch and Kibana
3. Install HDP Search 
(https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_solr-search-installation/content/ch_hdp-search-install-ambari.html)
4. Create collections for bro, snort and error using the 
`$METRON_HOME/bin/create_collection.sh` script.
5. Kerberize full dev using the instructions in 
https://github.com/apache/metron/blob/master/metron-deployment/Kerberos-manual-setup.md
6. In Ambari, change the "Solr Zookeeper Urls" setting in the Metron > 
Index Settings tab to "node1:2181/solr" and the "Random Access Search Engine" 
setting in the Metron > Indexing tab to "Solr" (it also helps to change "Random 
Access Indexing Offset" in the Metron > Indexing tab to "LATEST")
7. Verify data is showing up in Solr
8. Restart Metron REST and you should be able to query data in Solr without 
issue

This PR assumes HDP Search is being used.  The benefit of using HDP Search 
is that Ambari handles Kerberos configuration for Solr.  If using a separate 
Solr install, Kerberos configuration would need to be done manually there.

Side note:  I had a lot of trouble getting everything to work all at once 
due to resource constraints in full dev.  I would suggest shutting down as many 
services as possible, getting data into Solr first, then shutting down 
topologies, then starting REST and querying data.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron solr-kerberos

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/960.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #960


commit ac1de170af8fa769683f72783f6258cf12663f94
Author: merrimanr 
Date:   

[DISCUSS] Development environment for UI work

[Ryan Merriman]

In anticipation of more UI work starting (Alerts UI specifically), I want
to kick off a discussion about how we can best provide a backend for UI
developers to code against.  I believe our full dev environment will not
work for this because:

   1. It has become more resource intensive over time and requires constant
   tuning to make it stable.
   2. Takes a long time to build.  I think this will eventually lead to UI
   developers building less often and not being current with master.
   3. UI developers will likely not have experience with big data projects
   and should not be required to manage a full metron VM install locally.

I think we need something more lightweight that only includes components
needed for UI development.  Here are what I think are the requirements for
this:

   1. Must include the REST component as this is how the UIs interact with
   Metron
   2. Should also include Search (ES or Solr), Kafka, HBase and Zookeeper
   since REST depends on these (am I missing any?).
   3. Should be portable such that it can be run locally or in a shared
   environment like AWS
   4. The environment should always contain the latest version of master

There is currently work being done to externalize our integration test
infrastructure (https://issues.apache.org/jira/browse/METRON-1352) that I
believe can also be leveraged here.  Are there other options or approaches
people can think of?  What's the best way to provide an environment that's
easy to spin up and always current with master?

[GitHub] metron pull request #959: METRON-1485 Upgrade vagrant for dev environments

[JonZeolla]

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron/pull/959

METRON-1485 Upgrade vagrant for dev environments

## Contributor Comments
It looks like we are going to be forced into upgrading vagrant based on 
some HashiCorp deprecation activities.  See:

https://www.hashicorp.com/blog/terraform-enterprise-saas-has-a-new-address

I'm still digging around for the vagrant commit that fixes this in newer 
versions - I just upgraded to the latest and that fixed it.

Currently, if you don't have centos6 or ubuntu14 locally, you get see 
something like the following:

$ vagrant up
Bringing machine 'node1' up with 'virtualbox' provider...
==> node1: Box 'centos/6' could not be found. Attempting to find and 
install...
node1: Box Provider: virtualbox
node1: Box Version: >= 0
The box 'centos/6' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:

URL: ["https://atlas.hashicorp.com/centos/6;]
Error: The requested URL returned error: 404 Not Found

## Testing
Testing is underway, just need to upgrade vagrant and spin up the dev 
environment to test.  Review for errors and such.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron METRON-1485

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/959.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #959


commit 4ca17d5780729a61615e7ee6bc86e6ddf86c339e
Author: Nick Allen 
Date:   2017-11-27T20:29:38Z

METRON-1320 Cannot perform a bare-metal installation

commit b64606997307e233a4ef9741c2ad16e622eac952
Author: Nick Allen 
Date:   2017-11-27T20:55:12Z

Cleaning up after the C++ file(s) that we create

commit 551e3084c60346a6459101994043afa84869ab61
Author: Jon Zeolla 

[GitHub] metron-bro-plugin-kafka pull request #5: METRON-1407: Metron-Bro-Kafka plugi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron-bro-plugin-kafka/pull/5


---

[GitHub] metron-bro-plugin-kafka pull request #7: METRON-1324: Increment metron-bro-p...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron-bro-plugin-kafka/pull/7


---

[GitHub] metron pull request #948: METRON-1468: Add support for apache/metron-bro-plu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/948


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/946
  
@cestella " Yeah, I think that's the approach, however, there's a snag. 
Storm requires us to create uber jars, so probably what we want to do is have 
users actually put the xpath transport client on the storm.library.path."  If 
only there was a way to load things into storm with classloader isolation and 
dependency inclusion.


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/946
  
would this have any effect on people using x-pack alternatives?


---

[GitHub] metron issue #952: METRON-1480 Add yarn as default build tool for the fronte...

[xyztdanid4]

Github user xyztdanid4 commented on the issue:

https://github.com/apache/metron/pull/952
  
@merrimanr 
As we agreed yesterday I did the required changes in the pom xml-s.
I had to update the node dependency of the projects, otherwise the build 
was failed, cause one of the dependencies was dependent on node8.
There is no equivalent command to npm --prefix in yarn, so I updated the 
readme to just cd into that directory where the install step is required.
When we build the metron-alert project (angular4), there was an error, 
mainly because angular4 uses AOT as default builder strategy. I fixed this by 
addig env=prod to the package json.


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/946
  
@ottobackwards in it's current state, sort of, but you're not required to 
turn it on. In the desired (reflection based nifi style state) no, it should 
load it and use it if present, but otherwise just use the vanilla transport 
client. Of course someday they're going to force us to REST, which will bring 
all this up again no doubt.


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/946
  
wait, does this PR mean we *require* x-pack from now on?


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r173416554
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
 ---
@@ -70,6 +70,11 @@ def build_global_config_patch(params, patch_file):
 "path": "/es.date.format",
 "value": "{{es_date_format}}"
 },
+{
+"op": "add",
+"path": "/es.xpack.user",
+"value": "{{es_xpack_user}}"
--- End diff --

could we lean on the hadoop credentials apis? 


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[wardbekker]

Github user wardbekker commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r173398330
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
 ---
@@ -70,6 +70,11 @@ def build_global_config_patch(params, patch_file):
 "path": "/es.date.format",
 "value": "{{es_date_format}}"
 },
+{
+"op": "add",
+"path": "/es.xpack.user",
+"value": "{{es_xpack_user}}"
--- End diff --

Good point. Is there an existing encryption mechanism that we can use to 
prevent plain text passwords here? 


---

[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

[wardbekker]

Github user wardbekker commented on the issue:

https://github.com/apache/metron/pull/946
  
@mmiklavc yes, correct, like with ES you need to install x-pack for Kibana. 
You will be prompted for username password after restart of Kibana. 


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[wardbekker]

Github user wardbekker commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r173397551
  
--- Diff: pom.xml ---
@@ -97,7 +97,7 @@
 
${base_hadoop_version}
 ${base_hbase_version}
 ${base_flume_version}
-5.6.2
+5.6.7
--- End diff --

There was no x-pack transport client build for 5.6.2


---