June 2023 08:19 PM
To: user@flink.apache.org
Cc: Shammon FY ; Kamal Mittal
Subject: Re: Flink bulk and record file source format metrices
Hi Kamal,
In a similar situation, when a decoding failure happened I would generate a
special record that I could then detect/filter out (and increment a
zjur...@gmail.com>>
> Sent: 14 June 2023 05:33 PM
> To: Kamal Mittal <mailto:kamal.mit...@ericsson.com>>
> Cc: user@flink.apache.org <mailto:user@flink.apache.org>
> Subject: Re: Flink bulk and record file source format metrices
>
> Hi Kamal,
>
>
Hello,
Any way-forward, please suggest.
Rgds,
Kamal
From: Kamal Mittal via user
Sent: 15 June 2023 10:39 AM
To: Shammon FY
Cc: user@flink.apache.org
Subject: RE: Flink bulk and record file source format metrices
Hello,
I need one counter matric for no. of corrupt records while decoding
way “SourceReaderBase” class maintaining one counter
for no. of records emitted.
Rgds,
Kamal
From: Shammon FY
Sent: 14 June 2023 05:33 PM
To: Kamal Mittal
Cc: user@flink.apache.org
Subject: Re: Flink bulk and record file source format metrices
Hi Kamal,
Can you give more information about the
Hi Kamal,
Can you give more information about the metris you want? In Flink each
source task has one source reader which already has some metrics, you can
refer to metrics doc[1] for more detailed information.
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/
Best,
Shamm
Hello,
Using Flink record stream format file source API as below for parquet records
reading.
FileSource.FileSourceBuilder source =
FileSource.forRecordStreamFormat(streamformat, path);
source.monitorContinuously(Duration.ofMillis(1));
Want to log/generate metrices for corrupt records and
