Hi Karthi, Thanks for raising this proposal. It is a common use case to sink metric "data" into downstream Prometheus. The description in the motivation section is more or less misleading the discussion. I would suggest you rephrase it, e.g. metrics (pre)processing via Flink is ....
The current Flip does not contain too much information about how data will be sinked into Prometheus. Would you like to elaborate on it? Some architecture diagrams might help us better understand your thoughts. Just out of curiosity, since you are focusing on building the Prometheus connector, does it make sense to build the Prometheus Source too? Short-term stored metrics could be consumed by the Prometheus Source and, depending on requirements, might perform some processing like aggregation. Best regards, Jing On Wed, May 17, 2023 at 6:13 PM Danny Cranmer <dannycran...@apache.org> wrote: > Thanks for the FLIP. > > I agree that there is a real usecase for metrics sink vs metric reporter. > The metric reporters in Flink cover metrics about the Flink job, and a sink > is used when the metrics are the _data_. > > +1 on the FLIP ID, can you please fix that? > > With regard to this FLIP. > 1/ The fact we are using the remote write feature is not covered beyond the > code example. Can we add details on this to make it clear? Additionally > would this support _any_ Prometheus server or do we need to enable remote > endpoint feature on ther server? > 2/ Are there any concerns around Prometheus versioning or is the API > backwards compatible? Which versions of Prometheus will we be supporting? > 3/ With regard to the "AmazonPrometheusRequestSigner" the example has > static creds. Can we integrate with the AWS Util to support all credential > providers, static and dynamic? > > Thanks, > Danny > > On Wed, May 17, 2023 at 4:34 PM Teoh, Hong <lian...@amazon.co.uk.invalid> > wrote: > > > Thanks Karthi for creating the FLIP! > > > > Re: Martijn, > > > > As I understand it, the motivation for the Prometheus Sink is for users > > who want to write metrics to a Prometheus remote_write endpoint as an > > output of their job graph, so it would be good to treat it as a > first-class > > citizen and do it as part of Flink’s data flow. This way, we would > benefit > > from at least once guarantees, scaling, state management. > > > > For example, a user might want to read in logs, perform some aggregations > > and publish it into a metrics store for visualisation. This might be a > > great use-case for reducing the cardinality of metrics! > > > > I think you might be referring to the metrics of the Flink job itself > > (e.g. CPU / memory metrics). For this use case, I would agree that we > > should not use this sink but instead use our metric reporters. > > > > Regards, > > Hong > > > > > > > > > On 16 May 2023, at 03:37, Lijie Wang <wangdachui9...@gmail.com> wrote: > > > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > > > > > Hi Karthi, > > > > > > I think you are using a wrong FLIP id, the FLIP-229 has already be > > used[1]. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job > > > > > > Best, > > > Lijie > > > > > > Martijn Visser <martijnvis...@apache.org> 于2023年5月16日周二 04:44写道： > > > > > >> Hi Karthi, > > >> > > >> Thanks for the FLIP and opening up the discussion. My main question > is: > > why > > >> should we create a separate connector and not use and/or improve the > > >> existing integrations with Prometheus? I would like to understand more > > so > > >> that it can be added to the motivation of the FLIP. > > >> > > >> Best regards, > > >> > > >> Martijn > > >> > > >> On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan < > > kar...@karthitect.com> > > >> wrote: > > >> > > >>> Hello all, > > >>> > > >>> We would like to start a discussion thread on FLIP-229: Prometheus > Sink > > >>> Connector [1] where we propose to provide a sink connector for > > Prometheus > > >>> [2] based on the Async Sink [3]. Looking forward to comments and > > >> feedback. > > >>> Thank you. > > >>> > > >>> [1] > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector > > >>> [2] https://prometheus.io/ > > >>> [3] > > >>> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink > > >>> > > >> > > > > >
