Re: Raise alarm for corrupt records
Hi Kamal, No, but it should be straightforward to create metrics or events for these types of situations and integrate them with your own alerting solution. Best regards, Martijn On Wed, Jun 7, 2023 at 8:25 AM Kamal Mittal via user wrote: > Hello Community, > > > > Is there any way Flink provides out of box to raise alarm for corrupt > records (e.g. due to decoding failure) in between of running data pipeline > and send this alarm to outside of task manager process? > > > > Rgds, > > Kamal >
RE: Raise alarm for corrupt records
Hello, Thanks for quick reply. I am using parquet encoder/decoder and during decoding if any corrupt record comes then need to raise alarm and maintain metrices visible over Flink Metrices GUI. So any custom metrices can be created in Flink? Please give some reference of any such documentation. Rgds, Kamal From: Martijn Visser Sent: 07 June 2023 12:31 PM To: Kamal Mittal Cc: user@flink.apache.org Subject: Re: Raise alarm for corrupt records Hi Kamal, No, but it should be straightforward to create metrics or events for these types of situations and integrate them with your own alerting solution. Best regards, Martijn On Wed, Jun 7, 2023 at 8:25 AM Kamal Mittal via user mailto:user@flink.apache.org>> wrote: Hello Community, Is there any way Flink provides out of box to raise alarm for corrupt records (e.g. due to decoding failure) in between of running data pipeline and send this alarm to outside of task manager process? Rgds, Kamal
Re: Raise alarm for corrupt records
Hi Kamal, Documentation on the metrics can be found at https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/ Best regards, Martijn On Wed, Jun 7, 2023 at 10:13 AM Kamal Mittal via user wrote: > Hello, > > > > Thanks for quick reply. > > > > I am using parquet encoder/decoder and during decoding if any corrupt > record comes then need to raise alarm and maintain metrices visible over > Flink Metrices GUI. > > > > So any custom metrices can be created in Flink? Please give some reference > of any such documentation. > > > > Rgds, > > Kamal > > > > *From:* Martijn Visser > *Sent:* 07 June 2023 12:31 PM > *To:* Kamal Mittal > *Cc:* user@flink.apache.org > *Subject:* Re: Raise alarm for corrupt records > > > > Hi Kamal, > > > > No, but it should be straightforward to create metrics or events for these > types of situations and integrate them with your own alerting solution. > > > > Best regards, > > > > Martijn > > > > On Wed, Jun 7, 2023 at 8:25 AM Kamal Mittal via user < > user@flink.apache.org> wrote: > > Hello Community, > > > > Is there any way Flink provides out of box to raise alarm for corrupt > records (e.g. due to decoding failure) in between of running data pipeline > and send this alarm to outside of task manager process? > > > > Rgds, > > Kamal > >