[jira] [Created] (FLINK-33152) Prometheus Sink Connector - Integration tests
Lorenzo Nicora created FLINK-33152: -- Summary: Prometheus Sink Connector - Integration tests Key: FLINK-33152 URL: https://issues.apache.org/jira/browse/FLINK-33152 Project: Flink Issue Type: Sub-task Reporter: Lorenzo Nicora Integration tests against containerised Prometheus -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33141) Promentheus Sink Connector - Amazon Managed Prometheus Request Signer
Lorenzo Nicora created FLINK-33141: -- Summary: Promentheus Sink Connector - Amazon Managed Prometheus Request Signer Key: FLINK-33141 URL: https://issues.apache.org/jira/browse/FLINK-33141 Project: Flink Issue Type: Sub-task Components: Connectors / AWS Reporter: Lorenzo Nicora -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33142) Prometheus Sink Connector - Update Documentation
Lorenzo Nicora created FLINK-33142: -- Summary: Prometheus Sink Connector - Update Documentation Key: FLINK-33142 URL: https://issues.apache.org/jira/browse/FLINK-33142 Project: Flink Issue Type: Sub-task Components: Documentation Reporter: Lorenzo Nicora -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33140) Prometheus Sink Connector - E2E test
Lorenzo Nicora created FLINK-33140: -- Summary: Prometheus Sink Connector - E2E test Key: FLINK-33140 URL: https://issues.apache.org/jira/browse/FLINK-33140 Project: Flink Issue Type: Sub-task Reporter: Lorenzo Nicora -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33139) Prometheus Sink Connector - Table API support
Lorenzo Nicora created FLINK-33139: -- Summary: Prometheus Sink Connector - Table API support Key: FLINK-33139 URL: https://issues.apache.org/jira/browse/FLINK-33139 Project: Flink Issue Type: Sub-task Reporter: Lorenzo Nicora -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33138) Prometheus Connector Sink - DataStream API implementation
Lorenzo Nicora created FLINK-33138: -- Summary: Prometheus Connector Sink - DataStream API implementation Key: FLINK-33138 URL: https://issues.apache.org/jira/browse/FLINK-33138 Project: Flink Issue Type: Sub-task Reporter: Lorenzo Nicora -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33137) FLIP-312: Prometheus Sink Connector
Lorenzo Nicora created FLINK-33137: -- Summary: FLIP-312: Prometheus Sink Connector Key: FLINK-33137 URL: https://issues.apache.org/jira/browse/FLINK-33137 Project: Flink Issue Type: New Feature Reporter: Lorenzo Nicora Umbrella Jira for implementation of Prometheus Sink Connector https://cwiki.apache.org/confluence/display/FLINK/FLIP-312:+Prometheus+Sink+Connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[RESULT][VOTE] FLIP-312: Prometheus Sink Connector
Hello The proposal FLIP-312: Prometheus Sink Connector, has been approved with 8 votes (6 binding) Thanks everybody Lorenzo
Re: [DISCUSS] FLIP-229: Prometheus Sink Connector
Hi Danny this is a good idea. We can do it easily. I am amending the FLIP accordingly Regards Lorenzo On Mon, 18 Sept 2023 at 20:17, Danny Cranmer wrote: > Hello Lorenzo, > > > Please consider this is not an AWS-specific connector and will not depend > on flink-connector-aws-base > > This is my point, how can we introduce AWS specific functionality without > coupling them. How about this. The Prometheus connector does not depend on > aws-base but publishes a basic API for signing (that does not require a > dependency the other way around). Then we put the AWS signer in the AWS > repo, without taking a dependency on Prometheus. Thus the connectors are > completely decoupled and the core Prometheus repo does not contain any AWS > pollution? > > Danny > > On Mon, 18 Sept 2023, 18:16 Lorenzo Nicora, > wrote: > > > Hi Danny > > > > Please consider this is not an AWS-specific connector and will not depend > > on flink-connector-aws-base. > > Prometheus remote-write specs deem authentication as out-of-scope [1]. > > > > Amazon Managed Prometheus uses request signing to authenticate > > remote-writes. To support AMP but also keep the connector generic, the > idea > > is to define a genetic request signing interface, and provide an > > implementation for AMP. > > Also, to keep the connector API generic, we think the best approach is to > > pass a signer instance to the sink builder. > > > > I am not aware of Prometheus remote-write authentication schemes, other > > than request signing. > > The actual open questions are, in my opinion: > > 1. should we add the interface for other authentications. If so, what > > schemes? > > 2. should the AMP request signer be part of the connector or distributed > as > > a separate dependency? > > > > Regards > > Lorenzo > > > > [1] https://prometheus.io/docs/concepts/remote_write_spec/#out-of-scope > > > > > > On Mon, 18 Sept 2023 at 17:22, Danny Cranmer > > wrote: > > > > > Thanks for the reply Lorenzo. > > > > > > > Static credentials are just for the sake of the example. The current > > > prototype implementation already uses > DefaultAWSCredentialsProviderChain > > > that supports static and dynamic credentials. We can make the > credential > > > provider configurable. > > > > > > The point here was that flink-connector-aws-base provides a common way > to > > > define AWS credentials in config. It would be nice to reuse this > > mechanism > > > for consistency. I am wondering if we will reuse this approach, and if > so > > > how the dependency hierarchy will look? We had a similar discussion > > > regarding the redshift connector [1]. The FLIP uses > > > "AWS_ACCESS_KEY_ID"/"AWS_SECRET_ACCESS_KEY"/etc which look like the > > > constants in flink-connector-aws [2]? > > > > > > Thanks, > > > Danny > > > > > > [1] https://lists.apache.org/thread/fowhltc1n0tn4627ycwhrd5p8ds77lm8 > > > [2] > > > > > > > > > https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws-base/src/main/java/org/apache/flink/connector/aws/config/AWSConfigConstants.java#L91 > > > > > > On Mon, Sep 18, 2023 at 3:50 PM Lorenzo Nicora < > lorenzo.nic...@gmail.com > > > > > > wrote: > > > > > > > Hello > > > > > > > > I am also re-sending an old answer I sent on May 24th, that, for some > > > > reason, did not appear in the thread. > > > > -- > > > > Q1) The fact we are using the remote write feature is not covered > > beyond > > > > the code example. Can we add details on this to make it clear? > > > Additionally > > > > would this support _any_ Prometheus server or do we need to enable > > remote > > > > endpoint feature on the server? > > > > > > > > A1) We use the remote-write API. The server must provide a standard > > > > remote-write endpoint. Remote-write specs do not say anything about > > > > authentication. At the moment we are planning to support 1/ > > > unauthenticated > > > > requests, 2/ AWS signed requests for AMP. The idea is the signer > > > interface > > > > allows transformations of the request URL. Request payload cannot be > > > > modified and must be Protobuf, as by spec. > > > > > > > > > > > > > > > > Q2) Are there any concerns a
Re: [DISCUSS] FLIP-229: Prometheus Sink Connector
Hi Danny Please consider this is not an AWS-specific connector and will not depend on flink-connector-aws-base. Prometheus remote-write specs deem authentication as out-of-scope [1]. Amazon Managed Prometheus uses request signing to authenticate remote-writes. To support AMP but also keep the connector generic, the idea is to define a genetic request signing interface, and provide an implementation for AMP. Also, to keep the connector API generic, we think the best approach is to pass a signer instance to the sink builder. I am not aware of Prometheus remote-write authentication schemes, other than request signing. The actual open questions are, in my opinion: 1. should we add the interface for other authentications. If so, what schemes? 2. should the AMP request signer be part of the connector or distributed as a separate dependency? Regards Lorenzo [1] https://prometheus.io/docs/concepts/remote_write_spec/#out-of-scope On Mon, 18 Sept 2023 at 17:22, Danny Cranmer wrote: > Thanks for the reply Lorenzo. > > > Static credentials are just for the sake of the example. The current > prototype implementation already uses DefaultAWSCredentialsProviderChain > that supports static and dynamic credentials. We can make the credential > provider configurable. > > The point here was that flink-connector-aws-base provides a common way to > define AWS credentials in config. It would be nice to reuse this mechanism > for consistency. I am wondering if we will reuse this approach, and if so > how the dependency hierarchy will look? We had a similar discussion > regarding the redshift connector [1]. The FLIP uses > "AWS_ACCESS_KEY_ID"/"AWS_SECRET_ACCESS_KEY"/etc which look like the > constants in flink-connector-aws [2]? > > Thanks, > Danny > > [1] https://lists.apache.org/thread/fowhltc1n0tn4627ycwhrd5p8ds77lm8 > [2] > > https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws-base/src/main/java/org/apache/flink/connector/aws/config/AWSConfigConstants.java#L91 > > On Mon, Sep 18, 2023 at 3:50 PM Lorenzo Nicora > wrote: > > > Hello > > > > I am also re-sending an old answer I sent on May 24th, that, for some > > reason, did not appear in the thread. > > -- > > Q1) The fact we are using the remote write feature is not covered beyond > > the code example. Can we add details on this to make it clear? > Additionally > > would this support _any_ Prometheus server or do we need to enable remote > > endpoint feature on the server? > > > > A1) We use the remote-write API. The server must provide a standard > > remote-write endpoint. Remote-write specs do not say anything about > > authentication. At the moment we are planning to support 1/ > unauthenticated > > requests, 2/ AWS signed requests for AMP. The idea is the signer > interface > > allows transformations of the request URL. Request payload cannot be > > modified and must be Protobuf, as by spec. > > > > > > > > Q2) Are there any concerns around Prometheus versioning or is the API > > backwards compatible? Which versions of Prometheus will we be supporting > > > > A2) We are using the only version of Prometheus Remote-Write specs > > available v1.0, defined in Remote-Write spec document [1] published April > > 2023. There was a previous v0.1 draft version of the same specs. We will > > probably also be compatible with the draft version, but I still have to > > check the differences. > > > > > > > > Q3) With regard to the "AmazonPrometheusRequestSigner" the example has > > static creds. Can we integrate with the AWS Util to support all > credential > > providers, static and dynamic? > > > > A3) Static credentials are just for the sake of the example. The current > > prototype implementation already uses DefaultAWSCredentialsProviderChain > > that supports static and dynamic credentials. We can make the credential > > provider configurable. > > > > Lorenzo > > > > [1] https://prometheus.io/docs/concepts/remote_write_spec/ > > [2] > > > > > https://docs.google.com/document/d/1LPhVRSFkGNSuU1fBd81ulhsCPR4hkSZyyBj1SZ8fWOM/edit > > Lorenzo > > > > > > On Sun, 17 Sept 2023 at 09:51, Ahmed Hamdy wrote: > > > > > Thanks Lorenzo, > > > Looking forward to the PRs. > > > Best Regards > > > Ahmed Hamdy > > > > > > > > > On Sat, 16 Sept 2023 at 06:27, Lorenzo Nicora < > lorenzo.nic...@gmail.com> > > > wrote: > > > > > > > Hello > > > > > > > > (apologies if this is a duplicate
Re: [DISCUSS] FLIP-229: Prometheus Sink Connector
Hello I am also re-sending an old answer I sent on May 24th, that, for some reason, did not appear in the thread. -- Q1) The fact we are using the remote write feature is not covered beyond the code example. Can we add details on this to make it clear? Additionally would this support _any_ Prometheus server or do we need to enable remote endpoint feature on the server? A1) We use the remote-write API. The server must provide a standard remote-write endpoint. Remote-write specs do not say anything about authentication. At the moment we are planning to support 1/ unauthenticated requests, 2/ AWS signed requests for AMP. The idea is the signer interface allows transformations of the request URL. Request payload cannot be modified and must be Protobuf, as by spec. Q2) Are there any concerns around Prometheus versioning or is the API backwards compatible? Which versions of Prometheus will we be supporting A2) We are using the only version of Prometheus Remote-Write specs available v1.0, defined in Remote-Write spec document [1] published April 2023. There was a previous v0.1 draft version of the same specs. We will probably also be compatible with the draft version, but I still have to check the differences. Q3) With regard to the "AmazonPrometheusRequestSigner" the example has static creds. Can we integrate with the AWS Util to support all credential providers, static and dynamic? A3) Static credentials are just for the sake of the example. The current prototype implementation already uses DefaultAWSCredentialsProviderChain that supports static and dynamic credentials. We can make the credential provider configurable. Lorenzo [1] https://prometheus.io/docs/concepts/remote_write_spec/ [2] https://docs.google.com/document/d/1LPhVRSFkGNSuU1fBd81ulhsCPR4hkSZyyBj1SZ8fWOM/edit Lorenzo On Sun, 17 Sept 2023 at 09:51, Ahmed Hamdy wrote: > Thanks Lorenzo, > Looking forward to the PRs. > Best Regards > Ahmed Hamdy > > > On Sat, 16 Sept 2023 at 06:27, Lorenzo Nicora > wrote: > > > Hello > > > > (apologies if this is a duplicate reply) > > > > I was working with Karthi on this connector, and I have taken over the > > development. > > We have a working version we would like to submit to the community. > > > > The renumbered FLIP-312 is also updated with more details [1]. > > Happy to answer any questions. > > > > Regards > > Lorenzo > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector > > > > On Mon, 21 Aug 2023, 13:06 Ahmed Hamdy, wrote: > > > > > Hello Karthi > > > Is this FLIP still in progress? I see the FLIP not updated & couldn't > > find > > > open JIRAs. > > > I am happy to take over if you are no longer working on this. > > > Best Regards > > > Ahmed Hamdy > > > > > > > > > On Mon, 22 May 2023 at 14:49, Martijn Visser > > > > wrote: > > > > > > > Hi all, > > > > > > > > > For example, a user might want to read in logs, perform some > > > aggregations > > > > and publish it into a metrics store for visualisation. This might be > a > > > > great use-case for reducing the cardinality of metrics! > > > > > > > > I can see that. What I would like to see in the FLIP is a proposal on > > the > > > > boundaries of the metrics reporter vs the Prometheus sink. I think > it's > > > > important that we make clear when to use a metric reporter and when > > not. > > > I > > > > can imagine that there will be Flink users who think that they can > get > > > data > > > > from the metric reporter, make aggregrations in Flink and then store > it > > > > using the Prometheus sink. > > > > > > > > Overall, I think more context must be added to the FLIP, especially > on > > > the > > > > motivation. > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > On Fri, May 19, 2023 at 4:28 PM Karthi Thyagarajan < > > > kar...@karthitect.com> > > > > wrote: > > > > > > > > > Hi Lijie > > > > > > > > > > Thank you for pointing this out. I've corrected it [1]. Also, this > > page > > > > > [2] still shows 178 and 229 as available, which is why I picked it > > up. > > > > > > > > > > Thanks > > > > > Karthi > > > > > > > > > > [1] > > > > > > > > &g
[VOTE] FLIP-312: Prometheus Sink Connector
Hi All, Thanks for the feedback on FLIP-312: Prometheus Sink Connector [1]. We updated the FLIP accordingly [2]. I would like to open the vote on FLIP-312. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1] https://lists.apache.org/thread/tm4qqfb4fxr7bc6nq5mwty1fqz8sj39x [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector Regards Lorenzo Nicora
Re: [DISCUSS] FLIP-229: Prometheus Sink Connector
Hello (apologies if this is a duplicate reply) I was working with Karthi on this connector, and I have taken over the development. We have a working version we would like to submit to the community. The renumbered FLIP-312 is also updated with more details [1]. Happy to answer any questions. Regards Lorenzo [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector On Mon, 21 Aug 2023, 13:06 Ahmed Hamdy, wrote: > Hello Karthi > Is this FLIP still in progress? I see the FLIP not updated & couldn't find > open JIRAs. > I am happy to take over if you are no longer working on this. > Best Regards > Ahmed Hamdy > > > On Mon, 22 May 2023 at 14:49, Martijn Visser > wrote: > > > Hi all, > > > > > For example, a user might want to read in logs, perform some > aggregations > > and publish it into a metrics store for visualisation. This might be a > > great use-case for reducing the cardinality of metrics! > > > > I can see that. What I would like to see in the FLIP is a proposal on the > > boundaries of the metrics reporter vs the Prometheus sink. I think it's > > important that we make clear when to use a metric reporter and when not. > I > > can imagine that there will be Flink users who think that they can get > data > > from the metric reporter, make aggregrations in Flink and then store it > > using the Prometheus sink. > > > > Overall, I think more context must be added to the FLIP, especially on > the > > motivation. > > > > Best regards, > > > > Martijn > > > > On Fri, May 19, 2023 at 4:28 PM Karthi Thyagarajan < > kar...@karthitect.com> > > wrote: > > > > > Hi Lijie > > > > > > Thank you for pointing this out. I've corrected it [1]. Also, this page > > > [2] still shows 178 and 229 as available, which is why I picked it up. > > > > > > Thanks > > > Karthi > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector > > > [2] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > > On May 15, 2023, at 9:37 PM, Lijie Wang > > wrote: > > > > > > > > > Hi Karthi, > > > > > > I think you are using a wrong FLIP id, the FLIP-229 has already be > > used[1]. > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job > > > > > > Best, > > > Lijie > > > > > > Martijn Visser 于2023年5月16日周二 04:44写道： > > > > > > Hi Karthi, > > > > > > Thanks for the FLIP and opening up the discussion. My main question is: > > why > > > should we create a separate connector and not use and/or improve the > > > existing integrations with Prometheus? I would like to understand more > so > > > that it can be added to the motivation of the FLIP. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan < > > kar...@karthitect.com> > > > wrote: > > > > > > > Hello all, > > > > > > > > We would like to start a discussion thread on FLIP-229: Prometheus > Sink > > > > Connector [1] where we propose to provide a sink connector for > > Prometheus > > > > [2] based on the Async Sink [3]. Looking forward to comments and > > > feedback. > > > > Thank you. > > > > > > > > [1] > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector > > > > [2] https://prometheus.io/ > > > > [3] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink > > > > > > > > > > > > > > > >
RE: Re: [DISCUSS] FLIP-229: Prometheus Sink Connector
Hello I was working with Karthi and took over development. I have a working version already. I also updated the FLIP (now FLIP-312, it was renumbered due to a clash) Please, let me know how to proceed with this. Happy to answer any questions about the FLIP Regards Lorenzo [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector On 2023/08/21 12:01:51 Ahmed Hamdy wrote: > Hello Karthi > Is this FLIP still in progress? I see the FLIP not updated & couldn't find > open JIRAs. > I am happy to take over if you are no longer working on this. > Best Regards > Ahmed Hamdy > > > On Mon, 22 May 2023 at 14:49, Martijn Visser > wrote: > > > Hi all, > > > > > For example, a user might want to read in logs, perform some aggregations > > and publish it into a metrics store for visualisation. This might be a > > great use-case for reducing the cardinality of metrics! > > > > I can see that. What I would like to see in the FLIP is a proposal on the > > boundaries of the metrics reporter vs the Prometheus sink. I think it's > > important that we make clear when to use a metric reporter and when not. I > > can imagine that there will be Flink users who think that they can get data > > from the metric reporter, make aggregrations in Flink and then store it > > using the Prometheus sink. > > > > Overall, I think more context must be added to the FLIP, especially on the > > motivation. > > > > Best regards, > > > > Martijn > > > > On Fri, May 19, 2023 at 4:28 PM Karthi Thyagarajan > > wrote: > > > > > Hi Lijie > > > > > > Thank you for pointing this out. I've corrected it [1]. Also, this page > > > [2] still shows 178 and 229 as available, which is why I picked it up. > > > > > > Thanks > > > Karthi > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector > > > [2] > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > > On May 15, 2023, at 9:37 PM, Lijie Wang > > wrote: > > > > > > > > > Hi Karthi, > > > > > > I think you are using a wrong FLIP id, the FLIP-229 has already be > > used[1]. > > > > > > [1] > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job > > > > > > Best, > > > Lijie > > > > > > Martijn Visser 于2023年5月16日周二 04:44写道： > > > > > > Hi Karthi, > > > > > > Thanks for the FLIP and opening up the discussion. My main question is: > > why > > > should we create a separate connector and not use and/or improve the > > > existing integrations with Prometheus? I would like to understand more so > > > that it can be added to the motivation of the FLIP. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan < > > kar...@karthitect.com> > > > wrote: > > > > > > > Hello all, > > > > > > > > We would like to start a discussion thread on FLIP-229: Prometheus Sink > > > > Connector [1] where we propose to provide a sink connector for > > Prometheus > > > > [2] based on the Async Sink [3]. Looking forward to comments and > > > feedback. > > > > Thank you. > > > > > > > > [1] > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector > > > > [2] https://prometheus.io/ > > > > [3] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink > > > > > > > > > > > > > > > >
Re: [DISCUSS] FLIP-229: Prometheus Sink Connector
Hi everybody I am working with Karthi on this connector. I think I can address some of the concerns raised by Danny et al. I do not have write access to the FLIP yet, but I will get it and do some edits. Also regarding the motivation, as suggested by Martijn. Q1) The fact we are using the remote write feature is not covered beyond the code example. Can we add details on this to make it clear? Additionally would this support _any_ Prometheus server or do we need to enable remote endpoint feature on the server? A1) We use the remote-write API. The server must provide a standard remote-write endpoint. Remote-write specs do not say anything about authentication. At the moment we are planning to support 1/ unauthenticated requests, 2/ AWS signed requests for AMP. The idea is the signer interface allows transformations of the request URL. Request payload cannot be modified and must be Protobuf, as by spec. Q2) Are there any concerns around Prometheus versioning or is the API backwards compatible? Which versions of Prometheus will we be supporting A2) We are using the only version of Prometheus Remote-Write specs available v1.0, defined in Remote-Write spec document [1] published April 2023. There was a previous v0.1 draft version of the same specs. We will probably also be compatible with the draft version, but I still have to check the differences. Q3) With regard to the "AmazonPrometheusRequestSigner" the example has static creds. Can we integrate with the AWS Util to support all credential providers, static and dynamic? A3) Static credentials are just for the sake of the example. The current prototype implementation already uses DefaultAWSCredentialsProviderChain that supports static and dynamic credentials. We can make the credential provider configurable. Lorenzo [1] https://prometheus.io/docs/concepts/remote_write_spec/ [2] https://docs.google.com/document/d/1LPhVRSFkGNSuU1fBd81ulhsCPR4hkSZyyBj1SZ8fWOM/edit On Wed, 24 May 2023 at 19:37, Nicora, Lorenzo wrote: > *From: *Martijn Visser > > *Subject: RE: [EXTERNAL][DISCUSS] FLIP-229: Prometheus Sink Connector* > > *Date: *22 May 2023 at 14:49:20 BST > > *To: * > > *Reply-To: * > > > > Hi all, > > > For example, a user might want to read in logs, perform some aggregations > > and publish it into a metrics store for visualisation. This might be a > great use-case for reducing the cardinality of metrics! > > I can see that. What I would like to see in the FLIP is a proposal on the > boundaries of the metrics reporter vs the Prometheus sink. I think it's > important that we make clear when to use a metric reporter and when not. I > can imagine that there will be Flink users who think that they can get data > from the metric reporter, make aggregrations in Flink and then store it > using the Prometheus sink. > > Overall, I think more context must be added to the FLIP, especially on the > motivation. > > Best regards, > > Martijn > > On Fri, May 19, 2023 at 4:28 PM Karthi Thyagarajan > wrote: > > > Hi Lijie > > Thank you for pointing this out. I've corrected it [1]. Also, this page > [2] still shows 178 and 229 as available, which is why I picked it up. > > Thanks > Karthi > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector > [2] > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > On May 15, 2023, at 9:37 PM, Lijie Wang wrote: > > > Hi Karthi, > > I think you are using a wrong FLIP id, the FLIP-229 has already be used[1]. > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job > > Best, > Lijie > > Martijn Visser 于2023年5月16日周二 04:44写道： > > Hi Karthi, > > Thanks for the FLIP and opening up the discussion. My main question is: why > should we create a separate connector and not use and/or improve the > existing integrations with Prometheus? I would like to understand more so > that it can be added to the motivation of the FLIP. > > Best regards, > > Martijn > > On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan > wrote: > > > Hello all, > > We would like to start a discussion thread on FLIP-229: Prometheus Sink > Connector [1] where we propose to provide a sink connector for Prometheus > [2] based on the Async Sink [3]. Looking forward to comments and > > feedback. > > Thank you. > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector > > [2] https://prometheus.io/ > [3] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink > > > > > > > > > Amazon Web Services EMEA SARL, 38 avenue John F. Kennedy, L-1855 > Luxembourg, R.C.S. Luxembourg B186284 > > Amazon Web Services EMEA Sarl, UK Branch, 1 Principal Place, Worship > Street, London, EC2A 2FA, United Kingdom, registered in England and Wales, > UK Establishment No. BR019315 > > >
[jira] [Created] (FLINK-18223) AvroSerializer does not correctly instantiate GenericRecord
Lorenzo Nicora created FLINK-18223:
--
Summary: AvroSerializer does not correctly instantiate
GenericRecord
Key: FLINK-18223
URL: https://issues.apache.org/jira/browse/FLINK-18223
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.10.1
Reporter: Lorenzo Nicora
{{AvroSerializer.createInstance()}} simply calls
{{InstantiationUtil.instantiate(type)}} to create a new instance, also when
type is GenericRecord.
This fails with an exception, because a GenericRecord must be instantiated
through {{GenericRecordBuilder}} but {{InstantiationUtil}} is not aware of it.
{code:java}
The class 'org.apache.avro.generic.GenericRecord' is not instantiable: The
class is not a proper class. It is either abstract, an interface, or a
primitive type.{code}
This can be proven with this test
{code:java}
@Test
public void shouldInstantiateGenericRecord() {
org.apache.avro.Schema SCHEMA = new
org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"Dummy\",\"namespace\":\"dummy\",\"fields\":[{\"name\":\"something\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}");
AvroSerializer serializer = new
AvroSerializer<>(GenericRecord.class, SCHEMA);
serializer.createInstance();
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17486) ClassCastException when copying AVRO SpecificRecord containing a decimal field
Lorenzo Nicora created FLINK-17486:
--
Summary: ClassCastException when copying AVRO SpecificRecord
containing a decimal field
Key: FLINK-17486
URL: https://issues.apache.org/jira/browse/FLINK-17486
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.10.0
Environment: Flink 1.10.0
AVRO 1.9.2
Java 1.8.0 (but also Java 14)
Scala binary 2.11
Reporter: Lorenzo Nicora
When consuming from a Kafka source AVRO SpecificRecord containing a {{decimal}}
(logical type) field, copying the record fails with:
{{java.lang.ClassCastException: class java.math.BigDecimal cannot be cast to
class java.nio.ByteBuffer}}
This code reproduces the problem:
{{AvroSerializer serializer = new AvroSerializer<>(Sample.class);}}
{{Sample s1 = Sample.newBuilder()}}
{{ .setPrice(BigDecimal.valueOf(42.32))}}
{{ .setId("A12345")}}
{{ .build();}}
{{Sample s2 = serializer.copy(s1);}}
The AVRO SpecificRecord is generated using avro-maven-plugin from this IDL:
{{@namespace("example.avro")}}
{{protocol SampleProtocol {}}
{{ record Sample{}}
{{ string id;}}
{{ decimal(9,2) price;}}
{{ timestamp_ms eventTime;}}
{{ }}}
{{}}}
The deepCopy of the record happens behind the scenes when attaching an
AssignerWithPeriodicWatermark to a Kafka Source consuming AVRO SpecificRecord
and using Confluent Schema Registry. The assigned extracts the event time from
the record and enabling bookmarking (not sure whether this is related)
A simplified version of the application is
[here|[https://github.com/nicusX/flink-avro-bug|https://github.com/nicusX/flink-avro-bug/blob/master/src/main/java/example/StreamJob.java]].
The problem looks similar to AVRO-1895 but that issue has been fixed since AVRO
1.8.2 (I'm using AVRO 1.9.2)
In fact, the following code doing deepCopy and only relying on AVRO does work:
{{Sample s1 = Sample.newBuilder()}}
{{ .setPrice(BigDecimal.valueOf(42.32))}}
{{ .setId("A12345")}}
{{ .build();}}
{{Sample s2 = Sample.newBuilder(s1).build();}}
A simplified version of the Flink application causing the problem is
[here|[https://github.com/nicusX/flink-avro-bug/blob/master/src/main/java/example/StreamJob.java]].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
