[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...
Github user cjolif commented on the issue: https://github.com/apache/flink/pull/5374 @tzulitai sure. done. ---
[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5374 @cjolif since we have now reached a conclusion on where the Elasticsearch connector should be improved in the future, could you maybe close this PR? I assume a new PR will be opened that subsumes this one. ---
[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...
Github user cjolif commented on the issue: https://github.com/apache/flink/pull/5374 good news the issue reported by @jdewald is now fixed in Elasticsearch master. Not sure which version of Elasticsearch will contain the fix though. ---
[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...
Github user jdewald commented on the issue: https://github.com/apache/flink/pull/5374 We were in the process of testing the Flink integration with the ES High Level Rest client in order to perform upgrades to ES6 and stumbled on some issues related to the bulk rejection retry handling. In short, swapping in the HLR backend will not perform retries correctly and it is not actually safe to simply call `add` on the `RequestIndexer` that is sent into the `ActionRequestFailureHander` from the Flink-side sink due to some synchronization issues. I have created a bug report that has been accepted on the Elasticsearch side https://github.com/elastic/elasticsearch/issues/28885 to automatically perform retries. One additional note however is that in the case where the retries have been used up from the `BackoffPolicy`, using the `RetryRejectionFailureHandler` appears to cause a deadlock, so this may be something to fix on the ElasticsearchSink side. But it may be necessary to independently confirm that in case that is an issue particular to my testing. (TL;DR: The RetryRejectionFailureHandler is not safe to use with HLR and I am not certain it's safe to use with the `TransportClient` based one either in the case that it actually triggers). ---
[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5374 @cjolif sorry, I didn't have the chance, yet. But I agree that we should try getting this in for 1.5 (I saw your reply on the 1.5 release discussion thread in the mailing lists). I'll try to get back to this as soon as possible. ---
[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...
Github user cjolif commented on the issue: https://github.com/apache/flink/pull/5374 @tzulitai did you have a chance to look at this? If you have any question please let me know? ---
[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...
Github user cjolif commented on the issue: https://github.com/apache/flink/pull/5374 thanks @tzulitai. Let me know if you have any question. ---
[GitHub] flink issue #5374: [FLINK-8101][flink-connectors] Elasticsearch 5.3+ (Transp...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5374 @cjolif Thanks a lot for the contribution! Iâll add reviewing the PR to my backlog. I would like / will try to take a look at the PR by the end of this week, after some 1.4.1 blockers which Iâm still busy with. ---
