GitHub user mmiklavc opened a pull request:
https://github.com/apache/metron/pull/1242
METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API
## Contributor Comments
https://issues.apache.org/jira/browse/METRON-1834
This task has been a long time coming after having completed the ES upgrade
in https://issues.apache.org/jira/browse/METRON-939. Motivation for completing
this now is that Elasticsearch will be deprecating use of the TransportClient
in v 7.x. This PR migrates the Elasticsearch client from TransportClient to the
newer Java REST API.
1.
https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/client.html
2.
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-overview.html
3.
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-level-migration.html
This builds off and finishes work started by @cestella here -
https://github.com/cestella/incubator-metron/tree/es_rest_client. I condensed
his branch into 1 flattened commit and built on top of it in order to provide
attribution.
I have a number of tasks I'm still working through, but I wanted to get the
review process started. I've minimally validated X-Pack auth and will have some
follow-up for SSL. Test plans and a breakdown of the changes will be soon to
follow. For starters, full dev should continue to work as normal and you should
see data flowing into indexes for bro, snort, and yaf. There are some
additional changes to how this client will be configured, which I'll be
documenting shortly. The new client does not take a Map of settings any longer
now that it is leverage Apache HTTP Async Client
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-low-usage-dependencies.html
under the hood. This meant choosing a set of properties to expose and doing a
translation to the builder pattern under the hood. Again, I'll have a write-up
of this in the migration guide and update the README's accordingly.
NOTE: This checks off 2 items from this follow-on list
https://github.com/apache/metron/pull/840#issuecomment-347281776
1. Fix Log4j logging problem - classpath issues
2. Migrate to new ES REST client
Per discussion in the Metron Slack channel, I will be updating the Jira
ticket with a series of tasks to be completed prior to acceptance, including
performance regression testing compared with the old API.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/metron es-rest-client
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1242.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1242
----
commit a7c7dc287b4f9c99c6780b934a0b6f433a03aa04
Author: cstella <cestella@...>
Date: 2018-10-09T00:06:52Z
Casey Stella - elasticsearch rest client migration base work
commit 10410ea9718a2a1b1d287fb4f22a6c98efb1fdaa
Author: Michael Miklavcic <michael.miklavcic@...>
Date: 2018-10-09T00:07:22Z
Update shade plugin version
commit a33a16872118175ed35729df6ddde2959e49ae2f
Author: Michael Miklavcic <michael.miklavcic@...>
Date: 2018-10-09T15:56:08Z
Fix es update dao test
commit 52c3c96d7657205e65d3bd0c0e35923a851911da
Author: Michael Miklavcic <michael.miklavcic@...>
Date: 2018-10-09T21:26:16Z
Merge with master. Fix es search integration tests
commit 4742832869f9512e11cab4a32109c15a2b17a92e
Author: Michael Miklavcic <michael.miklavcic@...>
Date: 2018-10-11T18:45:06Z
Merge branch 'master' into es-rest-client
commit 43809968320e586fd70411776140f3aa13a60195
Author: Michael Miklavcic <michael.miklavcic@...>
Date: 2018-10-11T23:59:25Z
Get shade plugin working with the new ES client and the
ClassIndexTransformer Shade plugin transformer.
commit af03f6f036e96c742db39733bf8ebc2cbf229129
Author: Michael Miklavcic <michael.miklavcic@...>
Date: 2018-10-19T02:56:52Z
Introduce config classes for managing ES client configuration. Translate
properties for new client.
commit dfcf9df2c9d41ddd777d02678c90fbb1e7dce1a3
Author: Michael Miklavcic <michael.miklavcic@...>
Date: 2018-10-19T19:21:23Z
Resolve merge conflicts with master
commit babe86451abe0f56a2cd5ed92e49cbed6bca31ae
Author: Michael Miklavcic <michael.miklavcic@...>
Date: 2018-10-19T19:50:31Z
Merge branch 'master' into es-rest-client
commit df2a77d15251d7c1ea77f87d238a0fba932ae4dd
Author: Michael Miklavcic <michael.miklavcic@...>
Date: 2018-10-19T20:19:50Z
Remove extra deps in elasticsearch pom
----
---
