Hi, I want to upgrade the version of flume-elasticsearch-sink elasticsearch.




At present, flume uses elasticsearch as a category of sink, but the currently 
supported version of elasticsearch is too low, 0.90.1, and the latest version 
has reached 8.x.x




At the same time, elasticsearch removed the type syntax between 6.x.x and 
7.x.x, which is a bit too weak for the existing flume-elasticsearch-sink 
component.




Based on this, I plan to upgrade the elasticsearch version here. The overall 
process generally includes the following processes:




1. List the existing flume-elasticsearch-sink configuration and supported 
functions, and write the schema characteristics of elasticsearch.

This is because the current stage uses tcp and http to write data, but in the 
higher version of elasticsearch, it may not be supported.




2. Use the Java Low Level REST Client provided by elasticsearch to realize the 
sink function.

The low level rest client provided by elasticsearch complies with the apache 
protocol, as shown in:

https://github.com/elastic/elasticsearch/tree/main/licenses

https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/java-rest-low-usage-maven.html

According to the official website, The low-level REST client is compatible with 
all Elasticsearch versions.

I have successfully written data to elasticsearch 0.90.1 through the 6.8.23 
version Http protocol

<dependency>

     <groupId>org.elasticsearch.client</groupId>

     <artifactId>elasticsearch-rest-client</artifactId>

     <version>6.8.23</version>

</dependency>




3. Version compatibility test.

Considering the existing version, it is necessary to conduct a compatibility 
test for each major version of elasticsearch in the past. Here, I understand 
that the main focus is on writing the results of the following versions

|
0.90.13
|
|
1.7.6
|
|
2.4.6
|
|
5.6.16
|
|
6.8.23
|
|
7.17.9
|
| 8.6.2 |




Finally, I hope you can give some suggestions and opinions, and hope that I can 
successfully complete the upgrade of elasticsearch, thank you.

Reply via email to