Re: IcebergSourceReader metrics
Hi Péter,
Yes, this is exactly what I was looking for. Thanks!
Chetas
On Thu, Mar 28, 2024 at 11:19 PM Péter Váry
wrote:
> Hi Chetas,
> Are you looking for this information?
>
> * public IcebergSourceReaderMetrics(MetricGroup metrics, String
> fullTableName) {*
> *MetricGroup readerMetrics =*
> *metrics.addGroup("IcebergSourceReader").addGroup("table",
> fullTableName);*
> *this.assignedSplits = readerMetrics.counter("assignedSplits");*
> *this.assignedBytes = readerMetrics.counter("assignedBytes");*
> *this.finishedSplits = readerMetrics.counter("finishedSplits");*
> *this.finishedBytes = readerMetrics.counter("finishedBytes");*
> *this.splitReaderFetchCalls =
> readerMetrics.counter("splitReaderFetchCalls");*
> * }*
>
>
> It could be found here:
>
> https://github.com/apache/iceberg/blob/main/flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergSourceReaderMetrics.java#L32-L39
>
> Added here:
> https://github.com/apache/iceberg/pull/5554
>
> I hope this helps,
> Peter
>
> Chetas Joshi ezt írta (időpont: 2024. márc. 29.,
> P, 2:43):
>
>> Hello,
>>
>> I am using Flink to read Iceberg (S3). I have enabled all the metrics
>> scopes in my FlinkDeployment as below
>>
>> metrics.scope.jm: flink.jobmanager
>> metrics.scope.jm.job: flink.jobmanager.job
>> metrics.scope.tm: flink.taskmanager
>> metrics.scope.tm.job: flink.taskmanager.job
>> metrics.scope.task: flink.task
>> metrics.scope.operator: flink.operator
>>
>>
>> I send these metrics to Datadog. I am specifically interested in the
>> IcebergSourceReader metrics. I could not find any information about what
>> metrics to expect here
>> <https://iceberg.apache.org/javadoc/1.1.0/org/apache/iceberg/flink/source/reader/IcebergSourceReaderMetrics.html>.
>> In datadog as well, I could not find any metrics related to the
>> IcebergTableSourceReader. Can someone help me understand what metrics
>> associated with the IcebergTableSourceReader should be reported and what
>> metricGroup (my guess was operator) should they be part of?
>>
>> Thank you
>> Chetas
>>
>
Re: IcebergSourceReader metrics
Hi Chetas,
Are you looking for this information?
* public IcebergSourceReaderMetrics(MetricGroup metrics, String
fullTableName) {*
*MetricGroup readerMetrics =*
*metrics.addGroup("IcebergSourceReader").addGroup("table",
fullTableName);*
*this.assignedSplits = readerMetrics.counter("assignedSplits");*
*this.assignedBytes = readerMetrics.counter("assignedBytes");*
*this.finishedSplits = readerMetrics.counter("finishedSplits");*
*this.finishedBytes = readerMetrics.counter("finishedBytes");*
*this.splitReaderFetchCalls =
readerMetrics.counter("splitReaderFetchCalls");*
* }*
It could be found here:
https://github.com/apache/iceberg/blob/main/flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergSourceReaderMetrics.java#L32-L39
Added here:
https://github.com/apache/iceberg/pull/5554
I hope this helps,
Peter
Chetas Joshi ezt írta (időpont: 2024. márc. 29.,
P, 2:43):
> Hello,
>
> I am using Flink to read Iceberg (S3). I have enabled all the metrics
> scopes in my FlinkDeployment as below
>
> metrics.scope.jm: flink.jobmanager
> metrics.scope.jm.job: flink.jobmanager.job
> metrics.scope.tm: flink.taskmanager
> metrics.scope.tm.job: flink.taskmanager.job
> metrics.scope.task: flink.task
> metrics.scope.operator: flink.operator
>
>
> I send these metrics to Datadog. I am specifically interested in the
> IcebergSourceReader metrics. I could not find any information about what
> metrics to expect here
> <https://iceberg.apache.org/javadoc/1.1.0/org/apache/iceberg/flink/source/reader/IcebergSourceReaderMetrics.html>.
> In datadog as well, I could not find any metrics related to the
> IcebergTableSourceReader. Can someone help me understand what metrics
> associated with the IcebergTableSourceReader should be reported and what
> metricGroup (my guess was operator) should they be part of?
>
> Thank you
> Chetas
>
IcebergSourceReader metrics
Hello, I am using Flink to read Iceberg (S3). I have enabled all the metrics scopes in my FlinkDeployment as below metrics.scope.jm: flink.jobmanager metrics.scope.jm.job: flink.jobmanager.job metrics.scope.tm: flink.taskmanager metrics.scope.tm.job: flink.taskmanager.job metrics.scope.task: flink.task metrics.scope.operator: flink.operator I send these metrics to Datadog. I am specifically interested in the IcebergSourceReader metrics. I could not find any information about what metrics to expect here <https://iceberg.apache.org/javadoc/1.1.0/org/apache/iceberg/flink/source/reader/IcebergSourceReaderMetrics.html>. In datadog as well, I could not find any metrics related to the IcebergTableSourceReader. Can someone help me understand what metrics associated with the IcebergTableSourceReader should be reported and what metricGroup (my guess was operator) should they be part of? Thank you Chetas
