GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/518
METRON-799: The MPack should function in a kerberized cluster
## Contributor Comments
Allows the Ambari Kerberos wizard to handle Metron setup.
Changes include:
- Creation of Keytabs
- Running everything as the Metron user, including Storm topology workers
(on a Kerberized cluster).
- Setup for Metron user to actually be able to run (client_jaas setup, home
Storm dir setup, etc.)
- Adjusting perms to 0755. The exception is the HDFS output folder on a
non-kerb cluster is left as 0775 because we don't have Storm running workers as
metron user on. When Kerberizing, the permissions will be restricted down to
0755.
- Kafka ACLs
- HBase ACLs
- Refactored Topic creation to use a common function so I didn't have to
edit the same thing 3 times.
- Automated updating of Storm configs (the AutoTGT and running workers as
user)
There's still more testing I want to do, but this is definitely far enough
along to submit a PR.
I've spun this up on full dev with the now modified Kerberos setup
instructions, with the caveat that Ambari's Storm service check fails (it's
harmless, as far as I can tell). See below for more details. As this does not
touch the sensors, data will need to be pushed manually (same as the old
instructions). I've been able to push data from Kafka to Elasticsearch/HDFS.
### The Bad News
I would love insight on a problem, if anybody has some. I haven't edited
the docs to reflect this yet, in the hopes it'll be resolved.
Storm's service check will fail during (and after) Kerberization. Metron
can immediately be started perfectly fine. Nothing is legit wrong, but this
setup means that the storm user is unable to submit to the cluster (it doesn't
have it's home directory setup with some configs). Unfortunately, Ambari runs
the service check as the storm user.
This can be worked around by creating ~storm/.storm/storm.yaml
```
nimbus.seeds : ['node1']
java.security.auth.login.config :
'/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf'
storm.thrift.transport :
'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin
```
`java.security.auth.login.conf` can also be
`/etc/storm/conf/storm_jaas.conf`, but the value above leads me to my next
point. All of these values already exist in storm.yaml. The fact that they
need to be specified again in the user's home is really strange. And it'll give
an error that the TGT found is not renewable, not something you'd expected.
I'm unsure if there are restrictions on where Ambari chooses to run service
check, so it's possible this would have to be setup on every node Storm lives
on the cluster. I'm also unsure if we can actually have Ambari automate this if
it turns out to be necessary, since we aren't the Storm service.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron (Incubating).
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- ~Have you written or updated unit tests and or integration tests to
verify your changes?~
- ~If adding new dependencies to the code, are these dependencies licensed
in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?~
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for
your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/incubator-metron mpack-kerberos
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/518.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #518
----
commit 7934bec0c8f302bba3f68b6afaaf89271b0efce5
Author: justinleet <justinjl...@gmail.com>
Date: 2017-03-15T18:55:06Z
hbase kinit for ambari
* hbase kinit
* Adding kinits around kafka topic creation
* adding in hbase acl
* All status checks, etc. seem to work now
* Default to PLAINTEXT
commit fe065f2ea39c3146a60fe23504655089309f5481
Author: justinleet <justinjl...@gmail.com>
Date: 2017-03-15T19:42:51Z
Splitting out ACLs from Kafka and HBase setup (#3)
commit ee8d35dc5dcdaee507ff906aa83e83033d0606ea
Author: justinleet <justinjl...@gmail.com>
Date: 2017-03-15T20:41:56Z
Kerb testing (#4)
* Splitting out ACLs from Kafka and HBase setup
* Moving user outside
commit 7c0c5588e1eb4100fe455c199b0729088f47546f
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-04-06T02:40:59Z
mostly working
commit f75b4c37e6f4c7e72ca5cdd22a4f8b830db3ecc3
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-04-06T12:49:14Z
working now again after some refactoring.
commit 91a89429201d3d45ece33dc3c36ee9c6a34ebbb6
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-04-06T18:59:21Z
fixes, refactoring, and improvements
commit fd5238f632a9d4fc62a3e59ca8d3bb24e2e3c55b
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-04-06T19:23:31Z
Cleanup
commit 569be68b17d0e84ece51015140e6eade755f3cb5
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-04-07T12:09:28Z
error topics plus fixes plus documentation
commit 5418a2eb914faac0450f7749b6c76f75a2da4e19
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-04-07T12:22:35Z
Removing leftover TODO
commit 049cf524f919f608dbce2b6c5dacbd4552ce3efb
Author: justinjleet <justinjl...@gmail.com>
Date: 2017-04-07T12:37:12Z
Caught a testing change in the last git add
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
