Hi Pablo, No worries at all!
Was wondering if “add any dependencies” means to add a side container to the dataflow runner? Like prometheus scraper? If so, is it only working for dataflow runner v2? We have not upgraded to v2 yet so it might be hard to do that in v1. Thanks so much! Siyu > On May 6, 2022, at 11:57 AM, Pablo Estrada <[email protected]> wrote: > > Sorry about the delay! > > Yes, you can add any dependencies to your image - and you can add custom > reporting of metrics that you're tracking directly. That may help? > > On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <[email protected] > <mailto:[email protected]>> wrote: > Hi Jeff, > > Thanks so much for your quick responses. It is unfortunate that histogram is > unavailable in dataflow. Do you know if there are any workaround? Or do you > think it is plausible if we can use runner v2 and customize the image with > Prometheus exporter? > > Thanks again! > Siyu > >> >> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <[email protected] >> <mailto:[email protected]>> wrote: >> >> >> Siyu - The Beam metrics interface includes the Distribution metric type >> which can be used for histograms: >> >> https://beam.apache.org/documentation/programming-guide/#types-of-metrics >> <https://beam.apache.org/documentation/programming-guide/#types-of-metrics> >> >> Particulars of support depend on the runner. For Cloud Dataflow, the >> reported values are MAX, MIN, MEAN, and COUNT, so no support for >> finer-grained percentiles: >> >> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics >> >> <https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics> >> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <[email protected] >> <mailto:[email protected]>> wrote: >> Hi Beam community, >> >> I am wondering if there is histogram metrics available (or alternative >> recommendations) for showing up quantiles. We have counter metrics already >> but we would also like to see some quantiles for different values. >> >> Thanks a lot! >> Siyu
