Hi Saisai, Thanks a lot for the link! This is exactly what I need. Just curious, why this PR has not been merged, as it seems to implement rather natural requirement.
There are a number or use cases which can benefit from this feature, e.g. - collecting business metrics based on the data's attributes and reporting them into the monitoring system as a side effect of the data processing - visualizing technical metrics by means of alternative software (e.g. grafana) - currently it's hardly possible to know the actual number of jobs, stages, tasks and their names and IDs in advance to register all the corresponding metrics statically. Kind Regards, Sergey On Mon, May 6, 2019, 16:07 Saisai Shao <sai.sai.s...@gmail.com> wrote: > I remembered there was a PR about doing similar thing ( > https://github.com/apache/spark/pull/18406). From my understanding, this > seems like a quite specific requirement, it may requires code change to > support your needs. > > Thanks > Saisai > > Sergey Zhemzhitsky <szh.s...@gmail.com> 于2019年5月4日周六 下午4:44写道： > >> Hello Spark Users! >> >> Just wondering whether it is possible to register a metric source without >> metrics known in advance and add the metrics themselves to this source >> later on? >> >> It seems that currently MetricSystem puts all the metrics from the >> source's MetricRegistry into a shared MetricRegistry of a MetricSystem >> during metric source registration [1]. >> >> So in case there is a new metric with a new name added to the source's >> registry after this source registration, then this new metric will not be >> reported to the sinks. >> >> What I'd like to achieve is to be able to register new metrics with new >> names dynamically using a single metric source. >> Is it somehow possible? >> >> >> [1] >> https://github.com/apache/spark/blob/51de86baed0776304c6184f2c04b6303ef48df90/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L162 >> >
