Have you considered setting the value for some of the series to a fixed value? For example, if you're not interested in the value for <task_id>, you could consider setting that to a fixed value "task_id" [1] ?
Best regards, Martijn [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#system-scope Op do 30 jun. 2022 om 15:52 schreef Weihua Hu <huweihua....@gmail.com>: > Hi, Filip > > You can modify the InfluxdbReporter code to rewrite the > notifyOfAddedMetric method and filter the required metrics for reporting. > > Best, > Weihua > > > On Thu, Jun 30, 2022 at 8:46 PM Filip Karnicki <filip.karni...@gmail.com> > wrote: > >> Hi All >> >> We're using the influx reporter (flink 1.14.3), which seems to create a >> series per: >> -[task|job]manager >> - host >> - job_id >> - job_name >> - subtask_index >> - task_attempt_id >> - task_attempt_num >> - task_id >> - tm_id >> >> which amounts to about 4k of series each time our job restarts itself >> >> We are currently experiencing problems with checkpoint duration timeouts >> (> 60s) (unrelated) and every 60 secs our job restarts and creates further >> 4k series in influxdb. >> >> Needless to say, the team managing influxdb is not too happy with the >> amount of series we create. >> >> Is there anything I can do to either reduce the number of series, or >> reduce the number of types of metrics in order to produce fewer series? (we >> don't view all the available metrics in grafana, so we don't necessarily >> have to send all of them) >> >> The db caps at 1M series, and with our current problems with >> checkpointing we go through that many in a matter of hours >> >> Many thanks >> Fil >> >>
