nicoloboschi opened a new pull request, #18577: URL: https://github.com/apache/pulsar/pull/18577
### Motivation In the ES Sink we use two different clients based on the target instance of elastic. With ES 7 the opensearch high level client is used and against ES 8 the newest Java client is used. With `stripNulls=false`, when sending json with null fields the first one correctly send the json keeping the field equals to null. The Java client instead drop the null fields during the payload serialization. This is useful for end users because Elastic let you to configure null_field for fields in order to be able to query the index checking for nulls. In any case, the usage of the client should be transparent to the user and it must behaves in the same way. ### Modifications * Upgraded client from 8.1.0 to 8.5.2 to get the latest fix about configuring JSON: https://github.com/elastic/elasticsearch-java/pull/417 * Configure the json mapper to retain null fields at client level. Note that the decision to keep or strip null fields is up to the sink itself with the option `stripNulls`. Therefore the client must be configured to always keep nulls. * Upgraded ES docker images to the latest ones (8.5.1 and 7.17.7) ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
