If it is non-compliant endpoint, the problem should appear in both
versions, isn't it? It is effecting more than one series. The set up is in
corporate org so I cannot expose end points publicly.
I have an prometheus front end instance that remote reads from multiple
prometheus backends. the time series is sharded across multiple backends.
The results are also inconsistent in 2.18.1. Sometimes I get fewer time
series back but what is consistent is the last data point is duplicated on
all time series. Just switching front end to 2.4 with same configuration
file fixes the problem.
On Saturday, June 20, 2020 at 4:43:32 AM UTC-4, Ben Kochie wrote:
>
> Yes, this sounds a lot like duplicate sample ingestion or some other
> non-compliant metrics endpoint.
>
> It would also be helpful to have the prometheus.yml configuration to
> understand what else is going on.
>
> On Sat, Jun 20, 2020 at 10:30 AM Julius Volz <juliu...@promlabs.com
> <javascript:>> wrote:
>
>> When I create a file "metrics" with your metrics from the 2.4 example
>> block and serve it to Prometheus (using just "python -m SimpleHTTPServer
>> 12345") and then query for "my_metric" from a 2.18 Prometheus, I don't see
>> this problem.
>>
>> Would it be feasible to share a minimal /metrics endpoint example that
>> reproduces this behavior in 2.18?
>>
>> On Sat, Jun 20, 2020 at 3:55 AM Johny <mailitt...@gmail.com <javascript:>>
>> wrote:
>>
>>> I recently upgraded Prometheus version 2.4 to 2.18. It seems there is
>>> some incompatibility in the api. When I query a metric, I get same value
>>> across all time series in 2.18.
>>>
>>> e.g.
>>> query: my_metric
>>>
>>> 2.4 api-
>>> Element Value
>>> my_metric{l1="a1",l2="b1",l3="x-y-z1"} 3434
>>> my_metric{l1="a2",l2="b2",l3="x-y-z2"} 3.433
>>> my_metric{l1="a3",l2="b3",l3="x-y-z3"} 94344
>>> my_metric{l1="a4",l2="b4",l3="x-y-z4"} 1000
>>>
>>> 2.18 api returns the same value for all time series-
>>> Element Value
>>> my_metric{l1="a1",l2="b1",l3="x-y-z1"} 1000
>>> my_metric{l1="a2",l2="b2",l3="x-y-z2"} 1000
>>> my_metric{l1="a3",l2="b3",l3="x-y-z3"} 1000
>>> my_metric{l1="a4",l2="b4",l3="x-y-z4"} 1000
>>>
>>> The same problem occurs when I query across time. If I plot
>>> count(my_metric) on the graph, I get a flat line in 2.18 whereas 2.4
>>> changes over time.
>>>
>>> However, when I query a single time series, I don't have this issue in
>>> 2.18.
>>>
>>> Is this a known incompatibility issue between the two versions and does
>>> it affect storage layer as well?
>>> How do I solve it?
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com <javascript:>.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/191deffd-faf7-4370-a907-15863e84f64ao%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/prometheus-users/191deffd-faf7-4370-a907-15863e84f64ao%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>>
>> --
>> Julius Volz
>> PromLabs - promlabs.com
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to promethe...@googlegroups.com <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/CAObpH5wg9q8S8yYUFs9cP5Ctoir5hqQ%3DVo4XMJt05WW6iKXX-w%40mail.gmail.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/CAObpH5wg9q8S8yYUFs9cP5Ctoir5hqQ%3DVo4XMJt05WW6iKXX-w%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/5f49b5ab-d28e-4574-81fd-3a300cbb2c70o%40googlegroups.com.
