[ https://issues.apache.org/jira/browse/IMPALA-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-8946. ----------------------------------- Fix Version/s: Impala 3.4.0 Resolution: Fixed > Prometheus histograms do not follow conventions > ----------------------------------------------- > > Key: IMPALA-8946 > URL: https://issues.apache.org/jira/browse/IMPALA-8946 > Project: IMPALA > Issue Type: Bug > Components: Backend > Affects Versions: Impala 3.3.0 > Reporter: Guillem > Assignee: Guillem > Priority: Minor > Labels: observability > Fix For: Impala 3.4.0 > > Attachments: > 0001-IMPALA-8946-Fix-histogram-rendering-to-Prometheus.patch, > 0001-IMPALA-8946-Fix-histogram.patch > > > We've been using Prometheus metrics and we've found that some standard > Prometheus parser can not properly interpret histograms from Impala. > For example, Python official client > ([https://github.com/prometheus/client_python)] can not properly read them. > I've been digging a little bit why it can't read them and I've found that > Impala does not adhere to textual histogram conventions. > The following link describes the conventions for rendering histograms on > Prometheus textual format: > [https://prometheus.io/docs/instrumenting/exposition_formats/#histograms-and-summaries] > This is an example of a rendered histogram on Impala 3.3 on Prometheus > endpoint: > {code:java} > # HELP impala_thrift_server_backend_svc_thread_wait_time Amount of time > clients of Impala Backend Server spent waiting for service threads > # TYPE impala_thrift_server_backend_svc_thread_wait_time histogram > impala_thrift_server_backend_svc_thread_wait_time{le="0.2"} 0 > impala_thrift_server_backend_svc_thread_wait_time{le="0.5"} 0 > impala_thrift_server_backend_svc_thread_wait_time{le="0.7"} 0 > impala_thrift_server_backend_svc_thread_wait_time{le="0.9"} 0 > impala_thrift_server_backend_svc_thread_wait_time{le="0.95"} 0 > impala_thrift_server_backend_svc_thread_wait_time{le="0.999"} 0 > impala_thrift_server_backend_svc_thread_wait_time_max 0 > impala_thrift_server_backend_svc_thread_wait_time_min 0 > impala_thrift_server_backend_svc_thread_wait_time_count 49 > {code} > The linked histogram conventions say that > {quote}Each bucket count of a histogram named x is given as a separate sample > line with the name x_bucket and a label \{le="y"} (where y is the upper bound > of the bucket). > {quote} > And also > {quote}A histogram must have a bucket with \{le="+Inf"}. Its value must be > identical to the value of x_count. > {quote} > The previous example should be formatted as: > {code:java} > # HELP impala_thrift_server_backend_svc_thread_wait_time Amount of time > clients of Impala Backend Server spent waiting for service threads > # TYPE impala_thrift_server_backend_svc_thread_wait_time histogram > impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.2"} 0 > impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.5"} 0 > impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.7"} 0 > impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.9"} 0 > impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.95"} 0 > impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.999"} 0 > impala_thrift_server_backend_svc_thread_wait_time_bucket{le="+Inf"} 49 > impala_thrift_server_backend_svc_thread_wait_time_count 49 > {code} > I've found that with this format, the official python client is able to > properly read the histograms. > Note also that metrics suffixed with `_min` and `_max` are also out of the > convention and they also break histogram parsing and maybe they need to be > reported as separated metrics (maybe as gauges?) > If you are fine with doing this changes, I already have a patch to improve > the histogram formatting and I can submit it to review. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org