GitHub user cjolif opened a pull request:
https://github.com/apache/flink/pull/5374
[FLINK-8101][flink-connectors] Elasticsearch 5.3+ (TransportClient) and 6.x
(RestHighLevelClient) support
## What is the purpose of the change
*The purpose of this PR is to add Elasticsearch 6.X support on top of the
RestHighLevelClient. Indeed TransportClient is now deprecated and will be
removed in 8.X. Also the hosted version of Elasticsearch often forbid the use
of TransportClient.*
## Brief change log
* First a set of changes are borrowed from #4675:
* Add createRequestIndex method in ElasticsearchApiCallBridge
* Add flink-connector-elasticsearch5.3 project
* Add BulkProcessorIndexer in connector-elasticsearch5.3 to convert
ActionRequest to DocWriteRequest
* Then on top of these changes and of being able to create a
RestHighLevelClient instead of TransportClient:
* Add createClient method in ElasticsearchApiCallBridge. As
TransportClient and RestClient have only the AutoCloseable interface in common,
this is what the method returns.
* Make ElasticsearchSinkBase agnostic to whether it is using a
TransportClient or RestClient by adding a createBulkProcessorBuilder method on
ElasticsearchApiCallBridge that the ElasticsearchSinkBase calls. Implement this
method on all bridges.
* Add flink-connector-elasticsearch6 project leveraging Rest Client while
all the other ones still use TransportClient.
## Verifying this change
This change added tests and can be verified as follows:
* Elasticsearch test base has also been reworked a little bit to leverage
it for testing the Rest client base implementation.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): if you use the
elasticsearch6 project, this adds dependencies on elasticsearch 6
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs & javadocs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cjolif/flink es53-es6
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5374.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5374
----
commit b6a2396b31ade29071c65efa72df9f8f1fab9af4
Author: zjureel <zjureel@...>
Date: 2017-09-15T03:51:35Z
[FLINK-7386] Change ElasticsearchApiCallBridge to abstract class and add
createRequestIndex method
commit 1e5b21a331dfaed50844e89986c313f5fc40bdbe
Author: zjureel <zjureel@...>
Date: 2017-09-15T03:55:16Z
[FLINK-7386] add flink-connector-elasticsearch5.3 for elasticsearch5.3 and
later versions
commit 5a6e840c316095dd4f65f559405b19dcda7a1ca0
Author: zjureel <zjureel@...>
Date: 2017-09-15T04:42:44Z
[FLINK-7386] add test case for ES53
commit 574818f0f56f6a2b644e271a05a0796d90598aef
Author: zjureel <zjureel@...>
Date: 2017-09-15T05:33:43Z
[FLINK-7386] add document for ES5.3
commit 14168825507ad98c49a63be8ceab23dc539ff977
Author: Christophe Jolif <cjolif@...>
Date: 2018-01-25T21:31:57Z
[FLINK-8101] Elasticsearch 6.X REST support
----
---
