Hi Mason, Thank you for the advice, as I tried, it works and reduces a lot in size.
On Fri, Jun 4, 2021 at 11:45 AM Mason Chen <mason.c...@apple.com> wrote: > Hi Kai, > > You can use the excluded variables config for the reporter. > > - metrics.reporter.<name>.scope.variables.excludes: (optional) A > semi-colon (;) separate list of variables that should be ignored by > tag-based reporters (e.g., Prometheus, InfluxDB). > > > > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/#reporter > > Best, > Mason > > On Jun 3, 2021, at 9:31 PM, Kai Fu <zzfu...@gmail.com> wrote: > > Hi team, > > We noticed that Prometheus metrics exporter exports all of the metrics at > the most fine-grained level, which is tremendous for the prometheus server > especially when the parallelism is high. The metrics volume crawled from a > single host(parallelism 8) is around 40MB for us currently. This is due to > *task_name > *attribute in the metrics generated by the engine being very long. The > task_name attribute is auto-generated from SQL job, and it seems it's > attaching all field names onto it. > > We want to reduce the metrics volume by either drop task_name or at some > more coarse-grained level. But I cannot find any related documents about > this, any advice on that? > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/ > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#list-of-all-variables > > -- > *Best wishes,* > *- Kai* > > > -- *Best wishes,* *- Kai*
