Github user jdewald commented on the issue:
https://github.com/apache/flink/pull/5374
We were in the process of testing the Flink integration with the ES High
Level Rest client in order to perform upgrades to ES6 and stumbled on some
issues related to the bulk rejection retry handling. In short, swapping in the
HLR backend will not perform retries correctly and it is not actually safe to
simply call `add` on the `RequestIndexer` that is sent into the
`ActionRequestFailureHander` from the Flink-side sink due to some
synchronization issues.
I have created a bug report that has been accepted on the Elasticsearch
side https://github.com/elastic/elasticsearch/issues/28885 to automatically
perform retries.
One additional note however is that in the case where the retries have been
used up from the `BackoffPolicy`, using the `RetryRejectionFailureHandler`
appears to cause a deadlock, so this may be something to fix on the
ElasticsearchSink side. But it may be necessary to independently confirm that
in case that is an issue particular to my testing. (TL;DR: The
RetryRejectionFailureHandler is not safe to use with HLR and I am not certain
it's safe to use with the `TransportClient` based one either in the case that
it actually triggers).
---
