Hi Alexander, yes in the first iteration the use case is to get visibility on failed ES requests. Usually we expose metrics to count failures and integrate them into dashboards and setup alerting rules which fire in case they hit a certain threshold.
In not Flink based applications which index data into ES we also applied failure handlers which evaluated the kind of error and triggered different actions. For example there are errors from which you can recover (ConnectException because of some network issues) - in this case we retried the indexing request or eventually even forced a stop of the client until the failure was resolved. On the other hand there can be errors from which you can not recover. In our scenario this was the case for indexing requests with malformed data (mapping failures) - here we just ignored the failure and dropped the messages. I’m not sure if we want to move this logic as well to the Flink applications, but I just wanted to mention that there can be situations where you might want to control the behavior of your application depending on the failure. Best Regards, Lars > Am 03.12.2021 um 09:05 schrieb Alexander Preuß > <alexanderpre...@ververica.com>: > > Hi Lars, > > What is your use case for the failure handler, just collecting metrics? We > want to remove the configurable failure handler in the new Sink API > implementation of the Elasticsearch connector in Flink 1.15 because it can be > a huge footgun with regards to delivery guarantees. > > Best Regards, > Alexander > > On Thu, Dec 2, 2021 at 6:23 PM Lars Bachmann <lars.bachm...@posteo.de > <mailto:lars.bachm...@posteo.de>> wrote: > Hi David, > > Thanks for the reply. I think especially in an error/failure handler metrics > are important in order to have proper monitoring/alerting in such cases. > Would be awesome if this could be added to Flink at some point :). > > Regards, > > Lars > >> Am 02.12.2021 um 18:13 schrieb David Morávek <d...@apache.org >> <mailto:d...@apache.org>>: >> >> Hi Lars, >> >> quickly looking at the ES connector code, I think you're right and there is >> no way to do that :( In general I'd say that being able to expose metrics >> is a valid request. >> >> I can imagine having some kind of `RichActionRequestFailureHandler` with >> `{get|set}RuntimeContext` methods. More or less the same thing we already do >> with for example the `RichFunction`. This unfortunately requires some work >> on the Flink side. >> >> cc @Arvid >> >> On Thu, Dec 2, 2021 at 5:52 PM <lars.bachm...@posteo.de >> <mailto:lars.bachm...@posteo.de>> wrote: >> Hi, >> >> is there a way to expose custom metrics within an elasticsearch failure >> handler (ActionRequestFailureHandler)? To register custom metrics I need >> access to the runtime context but I don't see a way to access the >> context in the failure handler. >> >> Thanks and regards, >> >> Lars >
