On Sun, Jan 31, 2021 at 7:08 PM Johny <mailittoyami...@gmail.com> wrote:
> Please not that I am doing a remote read to fetch metrics in my rules. Per > my metrics, no issues with consistency in remote reads. Oh, that's interesting. Have you been able to reproduce it with just the local storage yet? What system are you remote-reading from? > On Sunday, January 31, 2021 at 1:04:08 PM UTC-5 Johny wrote: > >> Unfortunately, I am still seeing the issue even after removing all the >> filters! >> Almost 5% of time series are not recorded. I am writing to local disk >> with sufficient volume. No errors in log. >> Reproduced issue with a single recording rule. >> >> Prometheus version 2.17.2. Are there any known bugs causing this behavior? >> >> >> >> On Friday, January 29, 2021 at 5:04:10 PM UTC-5 Julien Pivotto wrote: >> >>> >>> Johny, >>> >>> Do you have new lines in your label value? >>> Can you try: >>> >>> label4=~"(?s:.+)" >>> >>> Are you using the latest release? >>> >>> Regards, >>> >>> On 29 Jan 22:54, Julius Volz wrote: >>> > Hi Johny, >>> > >>> > If this is indeed happening the way you describe it, that would be a >>> bug. >>> > The 'label4=~".+"' selector definitely should match any series that >>> have a >>> > value for label4. >>> > >>> > What Prometheus version are you using, and could you file a bug report >>> with >>> > more details (maybe a minimal reproducible test case?) at >>> > https://github.com/prometheus/prometheus/issues/new/choose? >>> > >>> > Regards, >>> > Julius >>> > >>> > On Fri, Jan 29, 2021 at 9:36 PM Johny <mailitt...@gmail.com> wrote: >>> > >>> > > It appears the issue is tied to the use of regex wild card filter in >>> > > experession >>> > > >>> > > expr: my_sample_label{label1="..",label4=~".+"} >>> > > >>> > > When I remove the 2nd label, issue goes away. >>> > > >>> > > expr: my_sample_label{label1=".."} >>> > > >>> > > THis is strange as all underlying series clearly have label4 values. >>> Why >>> > > is this failing then? >>> > > >>> > > >>> > > On Friday, January 29, 2021 at 3:17:29 PM UTC-5 Johny wrote: >>> > > >>> > >> I'm having a consistent problem wherein some recorded metrics are >>> missing >>> > >> randomly. >>> > >> >>> > >> The underlying expression evaluates to around 50-100 time series. >>> When I >>> > >> filter the expression to produce fewer number of time series >>> (5-10), then >>> > >> the recorded metrics are available and I don't see the issue. Is >>> there some >>> > >> sort of limit on time series in Prometheus per rule expression? >>> > >> >>> > >> 5 labels in time series. >>> > >> e.g. >>> > >> >>> > >> >>> my_sample_label{label1="test1_test1.test1",label2="32323",label3="abc",label4="d32938-3230-dd.dd.ss",label5="ksldkdls"} >>> >>> > >> >>> > >> I even tried on a prometheus node with a single rule and I see the >>> same >>> > >> pattern. >>> > >> >>> > >> any help appreciated. >>> > >> >>> > >> >>> > >> >>> > > -- >>> > > You received this message because you are subscribed to the Google >>> Groups >>> > > "Prometheus Users" group. >>> > > To unsubscribe from this group and stop receiving emails from it, >>> send an >>> > > email to prometheus-use...@googlegroups.com. >>> > > To view this discussion on the web visit >>> > > >>> https://groups.google.com/d/msgid/prometheus-users/7fc899f7-c0d4-4e17-8d65-d9a56ebc074fn%40googlegroups.com >>> > > < >>> https://groups.google.com/d/msgid/prometheus-users/7fc899f7-c0d4-4e17-8d65-d9a56ebc074fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> >>> > > . >>> > > >>> > >>> > >>> > -- >>> > Julius Volz >>> > PromLabs - promlabs.com >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to prometheus-use...@googlegroups.com. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/CAObpH5zn%3DfrZEzsB0p7faNMBqbEkbU5%2Bh4sXS-TpScRASO3b2w%40mail.gmail.com. >>> >>> >>> -- >>> Julien Pivotto >>> @roidelapluie >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c2188a54-baed-477a-9be6-e54f0c4240bbn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/c2188a54-baed-477a-9be6-e54f0c4240bbn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5wpT25hGW02uQMvHmzn-xXRX_zgi8DGJBYjPVKOyJvA5A%40mail.gmail.com.
