Re: [prometheus-users] Re: Https issue when using prometheus federation

2022-07-19 Thread Shi Yan 
oh, sorry, you mean all the metrics. It's about several seconds

real0m7.361s
user0m0.130s
sys0m0.999s

On Wednesday, July 20, 2022 at 1:03:36 AM UTC+10 Stuart Clark wrote:

> On 19/07/2022 14:51, Shi Yan wrote:
> > Thanks, Brian for helping look into it.
> >
> > Yes, in our setup, `another_prom_server` is deployed on the k8s 
> > cluster and it is behind an F5 ingress proxy, which terminates the TLS 
> > protocol. So we use HTTPS here.
> > And I've tried to add port 443 explicitly in the targets config, but 
> > the error is still the same.
> >
> > msg="Scrape failed" err="Get 
> > \"
> https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D\ 
> ": 
>
> > read tcp x.x.x.x:58342->y.y.y.y:443: read: connection reset by peer"
> >
> > While I can manually curl it with either
> >  > curl https://example.com
> >  Found
> >
> > or the one with the exact URL parameters from the error msg.
> >  > curl 
> > 'https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D'
> >  .# can get all the metrics correctly
> >
> How long does it take curl to respond with all the metrics?
>
> Could it be that it takes a while and your load balancer is configured 
> with a shorter timeout?
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ad51764e-f32a-488e-b84a-977516177f70n%40googlegroups.com.

Re: [prometheus-users] Re: Https issue when using prometheus federation

2022-07-19 Thread Shi Yan 
Hi Stuart

The curl actually gets a response immediately,  as we only need to get a 
small set of metrics.

The curl command time is as follows
real0m0.077s
user0m0.016s
sys0m0.018s

On Wednesday, July 20, 2022 at 1:03:36 AM UTC+10 Stuart Clark wrote:

> On 19/07/2022 14:51, Shi Yan wrote:
> > Thanks, Brian for helping look into it.
> >
> > Yes, in our setup, `another_prom_server` is deployed on the k8s 
> > cluster and it is behind an F5 ingress proxy, which terminates the TLS 
> > protocol. So we use HTTPS here.
> > And I've tried to add port 443 explicitly in the targets config, but 
> > the error is still the same.
> >
> > msg="Scrape failed" err="Get 
> > \"
> https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D\ 
> ": 
>
> > read tcp x.x.x.x:58342->y.y.y.y:443: read: connection reset by peer"
> >
> > While I can manually curl it with either
> >  > curl https://example.com
> >  Found
> >
> > or the one with the exact URL parameters from the error msg.
> >  > curl 
> > 'https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D'
> >  .# can get all the metrics correctly
> >
> How long does it take curl to respond with all the metrics?
>
> Could it be that it takes a while and your load balancer is configured 
> with a shorter timeout?
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/b102271a-bd72-4595-9a7a-16c2b8d14040n%40googlegroups.com.

Re: [prometheus-users] Re: Https issue when using prometheus federation

2022-07-19 Thread Shi Yan 
Thanks Julien, but adding enable_http2: false doesn't help. Still same 
error.

On Wednesday, July 20, 2022 at 12:20:50 AM UTC+10 Julien Pivotto wrote:

> Could you try adding http2_enable: false and see if there is an 
> improvement?
>
> Le mar. 19 juil. 2022, 15:51, Shi Yan  a écrit :
>
>> Thanks, Brian for helping look into it.
>>
>> Yes, in our setup, `another_prom_server` is deployed on the k8s cluster 
>> and it is behind an F5 ingress proxy, which terminates the TLS protocol. So 
>> we use HTTPS here.
>> And I've tried to add port 443 explicitly in the targets config, but the 
>> error is still the same.
>>
>> msg="Scrape failed" err="Get \"
>> https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D\ 
>> ":
>>  
>> read tcp x.x.x.x:58342->y.y.y.y:443: read: connection reset by peer"
>>
>> While I can manually curl it with either
>>  > curl https://example.com
>>  Found
>>
>> or the one with the exact URL parameters from the error msg.
>>  > curl '
>> https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D'
>>  .# can get all the metrics correctly
>>
>>
>> Cheers
>>  
>>
>>
>> On Tuesday, July 19, 2022 at 6:25:00 PM UTC+10 Brian Candler wrote:
>>
>>> Can you show the exact curl command line, with just the hostname 
>>> replaced with "example.com" ?
>>>
>>> Try:
>>>
>>>   - targets:
>>> - another_prom_server:9090
>>>
>>> or
>>>
>>>   - targets:
>>> - another_prom_server:443
>>>
>>> or whatever is appropriate.  (I note you set "scheme: https" - is that 
>>> correct? Is this prometheus running behind a reverse proxy or ingress 
>>> proxy, or configured with web.config to serve TLS?)
>>>
>>> On Tuesday, 19 July 2022 at 06:13:38 UTC+1 yansh...@gmail.com wrote:
>>>
 I am trying to configure the prometheus federation, but the target is 
 not up and the only error I can see is `read: connection reset by peer`

 The scrape_config I've added is as follows:

 - job_name: federate
   scrape_interval: 30s
   scrape_timeout: 15s
   scheme: https
   honor_labels: true
   metrics_path: "/federate"
   params:
 match[]:
 - '{job="jobname"}'
   static_configs:
   - targets:
 - another_prom_server

 But if I use `curl` command from this central prometheus server, it 
 works and can return the metrics correctly. 

 another_prom_server is the one deployed by kube-prometheus-stack helm 
 chart. Not sure what is the issue here? Could anyone help advise, thanks! 

>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/9304c742-68a9-427d-8449-ec6cf83e0f9an%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ab3a7596-4545-49a1-b742-3c6a6efa9d5fn%40googlegroups.com.

Re: [prometheus-users] Re: Https issue when using prometheus federation

2022-07-19 Thread Stuart Clark 

On 19/07/2022 14:51, Shi Yan wrote:

Thanks, Brian for helping look into it.

Yes, in our setup, `another_prom_server` is deployed on the k8s 
cluster and it is behind an F5 ingress proxy, which terminates the TLS 
protocol. So we use HTTPS here.
And I've tried to add port 443 explicitly in the targets config, but 
the error is still the same.


msg="Scrape failed" err="Get 
\"https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D\": 
read tcp x.x.x.x:58342->y.y.y.y:443: read: connection reset by peer"


While I can manually curl it with either
 > curl https://example.com
 Found

or the one with the exact URL parameters from the error msg.
 > curl 
'https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D'

 .# can get all the metrics correctly


How long does it take curl to respond with all the metrics?

Could it be that it takes a while and your load balancer is configured 
with a shorter timeout?


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e0c701db-9c73-becb-c9a6-743f6c7384b3%40Jahingo.com.

Re: [prometheus-users] Re: Https issue when using prometheus federation

2022-07-19 Thread Julien Pivotto 
Could you try adding http2_enable: false and see if there is an improvement?

Le mar. 19 juil. 2022, 15:51, Shi Yan  a écrit :

> Thanks, Brian for helping look into it.
>
> Yes, in our setup, `another_prom_server` is deployed on the k8s cluster
> and it is behind an F5 ingress proxy, which terminates the TLS protocol. So
> we use HTTPS here.
> And I've tried to add port 443 explicitly in the targets config, but the
> error is still the same.
>
> msg="Scrape failed" err="Get \"
> https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D\
> ":
> read tcp x.x.x.x:58342->y.y.y.y:443: read: connection reset by peer"
>
> While I can manually curl it with either
>  > curl https://example.com
>  Found
>
> or the one with the exact URL parameters from the error msg.
>  > curl '
> https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D'
>  .# can get all the metrics correctly
>
>
> Cheers
>
>
>
> On Tuesday, July 19, 2022 at 6:25:00 PM UTC+10 Brian Candler wrote:
>
>> Can you show the exact curl command line, with just the hostname replaced
>> with "example.com" ?
>>
>> Try:
>>
>>   - targets:
>> - another_prom_server:9090
>>
>> or
>>
>>   - targets:
>> - another_prom_server:443
>>
>> or whatever is appropriate.  (I note you set "scheme: https" - is that
>> correct? Is this prometheus running behind a reverse proxy or ingress
>> proxy, or configured with web.config to serve TLS?)
>>
>> On Tuesday, 19 July 2022 at 06:13:38 UTC+1 yansh...@gmail.com wrote:
>>
>>> I am trying to configure the prometheus federation, but the target is
>>> not up and the only error I can see is `read: connection reset by peer`
>>>
>>> The scrape_config I've added is as follows:
>>>
>>> - job_name: federate
>>>   scrape_interval: 30s
>>>   scrape_timeout: 15s
>>>   scheme: https
>>>   honor_labels: true
>>>   metrics_path: "/federate"
>>>   params:
>>> match[]:
>>> - '{job="jobname"}'
>>>   static_configs:
>>>   - targets:
>>> - another_prom_server
>>>
>>> But if I use `curl` command from this central prometheus server, it
>>> works and can return the metrics correctly.
>>>
>>> another_prom_server is the one deployed by kube-prometheus-stack helm
>>> chart. Not sure what is the issue here? Could anyone help advise, thanks!
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/9304c742-68a9-427d-8449-ec6cf83e0f9an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAFJ6V0o2N8FwQCYMiuVB6R33NZA_rAwcbPxy9wHsCdRi7dGExw%40mail.gmail.com.

[prometheus-users] Re: Https issue when using prometheus federation

2022-07-19 Thread Shi Yan 
Thanks, Brian for helping look into it.

Yes, in our setup, `another_prom_server` is deployed on the k8s cluster and 
it is behind an F5 ingress proxy, which terminates the TLS protocol. So we 
use HTTPS here.
And I've tried to add port 443 explicitly in the targets config, but the 
error is still the same.

msg="Scrape failed" err="Get 
\"https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D\": 
read tcp x.x.x.x:58342->y.y.y.y:443: read: connection reset by peer"

While I can manually curl it with either
 > curl https://example.com
 Found

or the one with the exact URL parameters from the error msg.
 > curl 
'https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D'
 .# can get all the metrics correctly


Cheers
 


On Tuesday, July 19, 2022 at 6:25:00 PM UTC+10 Brian Candler wrote:

> Can you show the exact curl command line, with just the hostname replaced 
> with "example.com" ?
>
> Try:
>
>   - targets:
> - another_prom_server:9090
>
> or
>
>   - targets:
> - another_prom_server:443
>
> or whatever is appropriate.  (I note you set "scheme: https" - is that 
> correct? Is this prometheus running behind a reverse proxy or ingress 
> proxy, or configured with web.config to serve TLS?)
>
> On Tuesday, 19 July 2022 at 06:13:38 UTC+1 yansh...@gmail.com wrote:
>
>> I am trying to configure the prometheus federation, but the target is not 
>> up and the only error I can see is `read: connection reset by peer`
>>
>> The scrape_config I've added is as follows:
>>
>> - job_name: federate
>>   scrape_interval: 30s
>>   scrape_timeout: 15s
>>   scheme: https
>>   honor_labels: true
>>   metrics_path: "/federate"
>>   params:
>> match[]:
>> - '{job="jobname"}'
>>   static_configs:
>>   - targets:
>> - another_prom_server
>>
>> But if I use `curl` command from this central prometheus server, it works 
>> and can return the metrics correctly. 
>>
>> another_prom_server is the one deployed by kube-prometheus-stack helm 
>> chart. Not sure what is the issue here? Could anyone help advise, thanks! 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/9304c742-68a9-427d-8449-ec6cf83e0f9an%40googlegroups.com.

[prometheus-users] Re: Https issue when using prometheus federation

2022-07-19 Thread Brian Candler 
Can you show the exact curl command line, with just the hostname replaced 
with "example.com" ?

Try:

  - targets:
- another_prom_server:9090

or

  - targets:
- another_prom_server:443

or whatever is appropriate.  (I note you set "scheme: https" - is that 
correct? Is this prometheus running behind a reverse proxy or ingress 
proxy, or configured with web.config to serve TLS?)

On Tuesday, 19 July 2022 at 06:13:38 UTC+1 yansh...@gmail.com wrote:

> I am trying to configure the prometheus federation, but the target is not 
> up and the only error I can see is `read: connection reset by peer`
>
> The scrape_config I've added is as follows:
>
> - job_name: federate
>   scrape_interval: 30s
>   scrape_timeout: 15s
>   scheme: https
>   honor_labels: true
>   metrics_path: "/federate"
>   params:
> match[]:
> - '{job="jobname"}'
>   static_configs:
>   - targets:
> - another_prom_server
>
> But if I use `curl` command from this central prometheus server, it works 
> and can return the metrics correctly. 
>
> another_prom_server is the one deployed by kube-prometheus-stack helm 
> chart. Not sure what is the issue here? Could anyone help advise, thanks! 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/29251b7e-d75e-46d9-b2a7-0cfd07010040n%40googlegroups.com.