GitHub user mmiklavc opened a pull request:
https://github.com/apache/metron/pull/840
METRON-939: Upgrade ElasticSearch and Kibana
DO NOT MERGE
This PR builds on the great work in
https://github.com/apache/metron/pull/619
## Contributor Comments
I wanted to get eyes on this despite some outstanding work left. It was no
small undertaking moving 3 major versions, so there's quite a bit to review. In
the future I would like to recommend we make feature branches for changes of
this magnitude.
This work upgrades Elasticsearch and Kibana both to 5.6. I'm running a
local full dev build now, but assuming no failures this will spin up full dev
with everything, including the Kibana dashboard (had to be completely
rewritten).
### TODO
- [ ] Another merge with master
- [ ] Update docs
- [ ] Docs for managing Kibana dashboards (no more pickle file - straight
JSON using ES's bulk load API. This was not as easy as you would think it would
be.)
- [ ] Test with Kerberos
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/metron elasticsearch562
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/840.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #840
----
commit 0805d20b95e3cdcf55bb0dfde91a08d8b9f58395
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-02T18:51:23Z
Version changes for MPack
commit 889c1bd5d24ee359d7d8113f500539ea10750a40
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-04T14:55:08Z
save demo code
commit 42f82180208dd7235214810406596bfe896665ab
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-17T12:12:43Z
Working through tests
commit 084db5ab4258a1dacb792ba504c1aeb18838327d
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-17T12:30:23Z
fix licensing
commit b33575c0b2131b7f44141342ebaaf3664fe6eab9
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-17T20:23:20Z
fix elasticsearchdao tests
commit 132b3330260dfe5a4b97e91d6210b381eda7222d
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-19T01:17:32Z
working through ES setting migrations for 2.x to 5.x
commit f71321485ea7b54be4ba45e969991bf3696c5197
Author: cstella <[email protected]>
Date: 2017-10-19T20:07:47Z
Updating to fix kibana mpack and log4j issue.
commit 54c2486e4e618a6d445667ed2040faacf165ca8c
Author: cstella <[email protected]>
Date: 2017-10-19T20:29:19Z
Removing argline..we should figure out why this is necessary, though. I
tried to up the version of surefire.
commit bd48988cf61344eeb2eebdd61caacf1ce83be1d5
Author: cstella <[email protected]>
Date: 2017-10-19T23:02:23Z
Avoiding log4j version mismatch.
commit 785e6a08558dec9e38468bc7ed50253f4e97ecdd
Author: cstella <[email protected]>
Date: 2017-10-20T16:43:01Z
Fixed some parameter name changes.
commit b8c81273de0fd82fa4c2978e8e62b3819173aaa7
Author: cstella <[email protected]>
Date: 2017-10-20T18:33:14Z
Removing and relocating old properties.
commit dcb2dbd1a96a31501fe4a0e0ffbdac70bc4e18b2
Author: cstella <[email protected]>
Date: 2017-10-20T21:38:28Z
Updating yamls.
commit d5629f0f53207012ee78c862d23c100431f8d2b1
Author: cstella <[email protected]>
Date: 2017-10-21T02:30:44Z
updating yaml.
commit d5e5cf7f9c353c45ce49b5fd47b2d37acf55d7c3
Author: cstella <[email protected]>
Date: 2017-10-21T02:51:11Z
Added a limits role to adjust security. We should consider doing this in
the mpack possibly..not sure how...
commit 67cc97d922ec06f231e45852dfc71a57618432c2
Author: cstella <[email protected]>
Date: 2017-10-21T03:20:46Z
typo
commit 297ed1d938892f5de48cc66caab013262be8cb4d
Author: cstella <[email protected]>
Date: 2017-10-21T04:22:41Z
forgot to remove bootstrap filter checking from master
commit 216b401382d43a9b7db0f9c8d3b410907e93810d
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-23T00:14:15Z
Merge pull request #11 from cestella/elasticsearch562_cstella
Day's work 10/19/2018 - 10/20/2018:)
commit fc92aa9edaacdfa3c618411a9f3f2426fcf51f1f
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-23T03:55:32Z
Get ES templates working
commit f3475bf7a50c733241f2aaf31dc03c08efcc89ad
Author: cstella <[email protected]>
Date: 2017-10-23T14:58:26Z
updating tests.
commit 12354430487f2a02fd50bc5ac2a6aa24c141dd6b
Author: cstella <[email protected]>
Date: 2017-10-23T16:21:01Z
Removing log4j-core.
commit 46e0a05b34e43fb2a42aa3ecb5f509678534ae79
Author: cstella <[email protected]>
Date: 2017-10-23T21:48:31Z
Updating poms to exclude the storm log4j.
commit 798f954ed6fb8598a27f37173a3f0ef66f0164b7
Author: cstella <[email protected]>
Date: 2017-10-23T23:47:43Z
Updating pom to include log4j
commit 9e2de3bf883b710a1c50f14f2fd4e119ff88e24a
Author: cstella <[email protected]>
Date: 2017-10-24T00:28:40Z
Adding back in teh log4j but with the right version.
commit c160837ae2863e71c5d392499af575ac1893fb0c
Author: cstella <[email protected]>
Date: 2017-10-24T13:42:55Z
Updating dependencies with url.
commit a134e230454db7d4dfff0ed8a308ab205efa4285
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-24T13:58:37Z
get netty conflicts with rest api working
commit 30795d78f4801bffdf50f52fa7aa566246f14e7c
Author: cstella <[email protected]>
Date: 2017-10-24T14:00:12Z
Merge remote-tracking branch 'mmiklavcic/elasticsearch562' into
elasticsearch562_tests
commit bdabc5b01f6f80f70394a1fa1088c512b2050ffe
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-24T14:35:52Z
Merge pull request #12 from cestella/elasticsearch562_tests
Testing fixes in travis.
commit e13688fbfcb09624fcad439959c285dd41e32a0b
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-24T17:12:54Z
Fix text/keyword types according to ES mapping recommendations
commit 193891fdee0ace19fde9f1c51ee7560c550c946b
Author: Michael Miklavcic <[email protected]>
Date: 2017-10-24T17:13:09Z
Merge branch 'elasticsearch562' of https://github.com/mmiklavc/metron into
elasticsearch562
commit feaa843ae745ddf66ab1f7504593326d09dda11f
Author: cstella <[email protected]>
Date: 2017-10-24T17:58:32Z
Merge branch 'master' into elasticsearch562_master
----
---
