Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[tejaswini vadlamudi]

Ben and Stuart, any comments?

On Thursday, June 2, 2022 at 1:53:49 PM UTC+2 tejaswini vadlamudi wrote:

> I'm not on top of performance degradation issues with the NFS storage 
> type. But by reading more on this topic, I understand the reasoning behind 
> your suggestion.
>
> NFS version 4 improved file locking.
>
> MySQL documentation says "If you choose to use NFS, NFS Version 4 or later 
> is recommended, as is testing your NFS setup thoroughly before deploying 
> into a production environment". 
>
> But Prometheus documentation and also the source code warn users to stay 
> away from it.
>
>
> https://github.com/prometheus/prometheus/blob/main/cmd/prometheus/main.go#L990
>
> case "NFS_SUPER_MAGIC":
>
>   level.Warn(logger).Log("fs_type", fsType, "msg", "This 
> filesystem is not supported and may lead to data corruption and data loss. 
> Please carefully read 
> https://prometheus.io/docs/prometheus/latest/storage/ to learn more about 
> supported filesystems.")
>
> What do you recommend if certain users have access to ONLY NFS storage 
> type? Do you suggest sending/streaming Prometheus data to someother long 
> term data store like Victoria Metrics or Cortext? I still wonder about 
> storage type for fsyncs and WAL file handling. 
> On K8S, if ephemeral storage is used to store Prometheus data,  data is 
> not persistent and highly available.
>
> Thanks!
> On Tuesday, May 31, 2022 at 9:37:12 PM UTC+2 sup...@gmail.com wrote:
>
>> Last I remember, most of the issues are around file locking and atomic 
>> operations.
>>
>> Agent mode with WAL-only should be less of an issue, except possibly 
>> fsyncs.
>>
>> Either way, depending on NFS is a bad idea from a reliability 
>> perspective, as your monitoring system now depends on the network and 
>> storage server to properly alert.
>>
>> Like Stuart says, would you run your SQL servers over NFS? 
>>
>> On Tue, May 31, 2022 at 7:38 PM tejaswini vadlamudi  
>> wrote:
>>
>>> What do you mean by not meeting desired performance characteristics? Is 
>>> it the write conflicts, file/block size limiations, etc.. eventually over 
>>> time, or is it with a high ingestion load?
>>> Do we have similar problems if we deploy Prometheus in Agent mode (data 
>>> streamed to a remote location) but still need WAL storage to handle 
>>> receiver connection failures?
>>>
>>> Thanks, Teja
>>> On Tuesday, May 31, 2022 at 5:41:11 PM UTC+2 Stuart Clark wrote:
>>>
 On 2022-05-31 15:45, tejaswini vadlamudi wrote: 
 > Hi Stuart, 
 > 
 > I forgot to ask the most important question on this topic :-) 
 > Could you explain the reason for not supporting NFS based storage in 
 > Prometheus? 
 > 

 At its core Prometheus contains a high performance time-series 
 database. 
 Network filesystems just don't have the same performance 
 characteristics 
 or features as a direct local disk. It is similar to not expecting 
 support for storing a MySQL/Oracle/MSSQL/etc. database on a network 
 storage system - it might technically work, but you are likely to 
 encounter problems at some point, and you are unlikely to recenive any 
 free/paid support in that architecture. 

 -- 
 Stuart Clark 

>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/7cfd5704-a10f-4007-92e2-7f2ffc21429dn%40googlegroups.com.

Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[Brian Candler]

On Thursday, 2 June 2022 at 18:51:56 UTC+1 jcour...@gmail.com wrote:

> How can I post a new question?
>

How did you post this one?

If it was by replying to an E-mail, then create a fresh E-mail to the same 
address, instead of replying to a message.

If it was using the Google Groups web interface, then click "+ New 
conversation".
 

> I have prometheus and alertmanager working separated and Alertmanager is 
> now using HTTPS (SSL certificate are installed).
> How can I tell prometheus to use HTTPS when it communicate with 
> AlertManager.
>

See 
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alertmanager_config

You need "scheme: https"

You may also need a "tls_config" block if you need to override any TLS 
settings, e.g. giving it a different ca_cert to validate against.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e0f349ba-2f81-4b05-86cc-688964a5873an%40googlegroups.com.

Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[Julio Cesar de la court]

Hallo, 
How can I post a new question?
I have prometheus and alertmanager working separated and Alertmanager is 
now using HTTPS (SSL certificate are installed).
How can I tell prometheus to use HTTPS when it communicate with 
AlertManager.
I think I need to change the following Prometheus setting:
 '''
 alertmanagers:
  - static_configs:
- targets:
  -  quimosabie.court.nl:9093
'''

Thank you in advance.

Julio
Op dinsdag 31 mei 2022 om 21:37:12 UTC+2 schreef sup...@gmail.com:

> Last I remember, most of the issues are around file locking and atomic 
> operations.
>
> Agent mode with WAL-only should be less of an issue, except possibly 
> fsyncs.
>
> Either way, depending on NFS is a bad idea from a reliability perspective, 
> as your monitoring system now depends on the network and storage server to 
> properly alert.
>
> Like Stuart says, would you run your SQL servers over NFS? 
>
> On Tue, May 31, 2022 at 7:38 PM tejaswini vadlamudi  
> wrote:
>
>> What do you mean by not meeting desired performance characteristics? Is 
>> it the write conflicts, file/block size limiations, etc.. eventually over 
>> time, or is it with a high ingestion load?
>> Do we have similar problems if we deploy Prometheus in Agent mode (data 
>> streamed to a remote location) but still need WAL storage to handle 
>> receiver connection failures?
>>
>> Thanks, Teja
>> On Tuesday, May 31, 2022 at 5:41:11 PM UTC+2 Stuart Clark wrote:
>>
>>> On 2022-05-31 15:45, tejaswini vadlamudi wrote: 
>>> > Hi Stuart, 
>>> > 
>>> > I forgot to ask the most important question on this topic :-) 
>>> > Could you explain the reason for not supporting NFS based storage in 
>>> > Prometheus? 
>>> > 
>>>
>>> At its core Prometheus contains a high performance time-series database. 
>>> Network filesystems just don't have the same performance characteristics 
>>> or features as a direct local disk. It is similar to not expecting 
>>> support for storing a MySQL/Oracle/MSSQL/etc. database on a network 
>>> storage system - it might technically work, but you are likely to 
>>> encounter problems at some point, and you are unlikely to recenive any 
>>> free/paid support in that architecture. 
>>>
>>> -- 
>>> Stuart Clark 
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/72583bb4-3bb0-4da5-a774-f76bf8831588n%40googlegroups.com.

Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[tejaswini vadlamudi]

I'm not on top of performance degradation issues with the NFS storage type. 
But by reading more on this topic, I understand the reasoning behind your 
suggestion.

NFS version 4 improved file locking.

MySQL documentation says "If you choose to use NFS, NFS Version 4 or later 
is recommended, as is testing your NFS setup thoroughly before deploying 
into a production environment". 

But Prometheus documentation and also the source code warn users to stay 
away from it.

https://github.com/prometheus/prometheus/blob/main/cmd/prometheus/main.go#L990

case "NFS_SUPER_MAGIC":

  level.Warn(logger).Log("fs_type", fsType, "msg", "This 
filesystem is not supported and may lead to data corruption and data loss. 
Please carefully read https://prometheus.io/docs/prometheus/latest/storage/ 
to learn more about supported filesystems.")

What do you recommend if certain users have access to ONLY NFS storage 
type? Do you suggest sending/streaming Prometheus data to someother long 
term data store like Victoria Metrics or Cortext? I still wonder about 
storage type for fsyncs and WAL file handling. 
On K8S, if ephemeral storage is used to store Prometheus data,  data is not 
persistent and highly available.

Thanks!
On Tuesday, May 31, 2022 at 9:37:12 PM UTC+2 sup...@gmail.com wrote:

> Last I remember, most of the issues are around file locking and atomic 
> operations.
>
> Agent mode with WAL-only should be less of an issue, except possibly 
> fsyncs.
>
> Either way, depending on NFS is a bad idea from a reliability perspective, 
> as your monitoring system now depends on the network and storage server to 
> properly alert.
>
> Like Stuart says, would you run your SQL servers over NFS? 
>
> On Tue, May 31, 2022 at 7:38 PM tejaswini vadlamudi  
> wrote:
>
>> What do you mean by not meeting desired performance characteristics? Is 
>> it the write conflicts, file/block size limiations, etc.. eventually over 
>> time, or is it with a high ingestion load?
>> Do we have similar problems if we deploy Prometheus in Agent mode (data 
>> streamed to a remote location) but still need WAL storage to handle 
>> receiver connection failures?
>>
>> Thanks, Teja
>> On Tuesday, May 31, 2022 at 5:41:11 PM UTC+2 Stuart Clark wrote:
>>
>>> On 2022-05-31 15:45, tejaswini vadlamudi wrote: 
>>> > Hi Stuart, 
>>> > 
>>> > I forgot to ask the most important question on this topic :-) 
>>> > Could you explain the reason for not supporting NFS based storage in 
>>> > Prometheus? 
>>> > 
>>>
>>> At its core Prometheus contains a high performance time-series database. 
>>> Network filesystems just don't have the same performance characteristics 
>>> or features as a direct local disk. It is similar to not expecting 
>>> support for storing a MySQL/Oracle/MSSQL/etc. database on a network 
>>> storage system - it might technically work, but you are likely to 
>>> encounter problems at some point, and you are unlikely to recenive any 
>>> free/paid support in that architecture. 
>>>
>>> -- 
>>> Stuart Clark 
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/43835db9-7c20-49eb-8890-a54a13dfc033n%40googlegroups.com.

Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[Ben Kochie]

Last I remember, most of the issues are around file locking and atomic
operations.

Agent mode with WAL-only should be less of an issue, except possibly fsyncs.

Either way, depending on NFS is a bad idea from a reliability perspective,
as your monitoring system now depends on the network and storage server to
properly alert.

Like Stuart says, would you run your SQL servers over NFS?

On Tue, May 31, 2022 at 7:38 PM tejaswini vadlamudi 
wrote:

> What do you mean by not meeting desired performance characteristics? Is it
> the write conflicts, file/block size limiations, etc.. eventually over
> time, or is it with a high ingestion load?
> Do we have similar problems if we deploy Prometheus in Agent mode (data
> streamed to a remote location) but still need WAL storage to handle
> receiver connection failures?
>
> Thanks, Teja
> On Tuesday, May 31, 2022 at 5:41:11 PM UTC+2 Stuart Clark wrote:
>
>> On 2022-05-31 15:45, tejaswini vadlamudi wrote:
>> > Hi Stuart,
>> >
>> > I forgot to ask the most important question on this topic :-)
>> > Could you explain the reason for not supporting NFS based storage in
>> > Prometheus?
>> >
>>
>> At its core Prometheus contains a high performance time-series database.
>> Network filesystems just don't have the same performance characteristics
>> or features as a direct local disk. It is similar to not expecting
>> support for storing a MySQL/Oracle/MSSQL/etc. database on a network
>> storage system - it might technically work, but you are likely to
>> encounter problems at some point, and you are unlikely to recenive any
>> free/paid support in that architecture.
>>
>> --
>> Stuart Clark
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmqms4_rJyOx3dm7E3jqC-n4U_CGSekNoK%3DU%2BqTP0aDuvA%40mail.gmail.com.

Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[tejaswini vadlamudi]

What do you mean by not meeting desired performance characteristics? Is it 
the write conflicts, file/block size limiations, etc.. eventually over 
time, or is it with a high ingestion load?
Do we have similar problems if we deploy Prometheus in Agent mode (data 
streamed to a remote location) but still need WAL storage to handle 
receiver connection failures?

Thanks, Teja
On Tuesday, May 31, 2022 at 5:41:11 PM UTC+2 Stuart Clark wrote:

> On 2022-05-31 15:45, tejaswini vadlamudi wrote:
> > Hi Stuart,
> > 
> > I forgot to ask the most important question on this topic :-)
> > Could you explain the reason for not supporting NFS based storage in
> > Prometheus?
> > 
>
> At its core Prometheus contains a high performance time-series database. 
> Network filesystems just don't have the same performance characteristics 
> or features as a direct local disk. It is similar to not expecting 
> support for storing a MySQL/Oracle/MSSQL/etc. database on a network 
> storage system - it might technically work, but you are likely to 
> encounter problems at some point, and you are unlikely to recenive any 
> free/paid support in that architecture.
>
> -- 
> Stuart Clark
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com.

Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[Stuart Clark]

On 2022-05-31 15:45, tejaswini vadlamudi wrote:

Hi Stuart,

I forgot to ask the most important question on this topic :-)
Could you explain the reason for not supporting NFS based storage in
Prometheus?



At its core Prometheus contains a high performance time-series database. 
Network filesystems just don't have the same performance characteristics 
or features as a direct local disk. It is similar to not expecting 
support for storing a MySQL/Oracle/MSSQL/etc. database on a network 
storage system - it might technically work, but you are likely to 
encounter problems at some point, and you are unlikely to recenive any 
free/paid support in that architecture.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/83d885d70a47b03fb24f22374c3d6618%40Jahingo.com.

Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[tejaswini vadlamudi]

Hi Stuart,

I forgot to ask the most important question on this topic :-) 
Could you explain the reason for not supporting NFS based storage in 
Prometheus?

Thanks,
Teja
On Wednesday, May 4, 2022 at 11:02:54 PM UTC+2 tejaswini vadlamudi wrote:

> Thanks Stuart, now I understand support in Prometheus project.
> Though renowned storage vendors claim full compliance and seem to work 
> with Prometheus data, it is better to stay away from it.
>
> Cheers, Teja
> On Tuesday, May 3, 2022 at 9:53:52 PM UTC+2 Stuart Clark wrote:
>
>> On 03/05/2022 01:34, tejaswini vadlamudi wrote:
>> > Prometheus documentation recommends to avoid NFS storage backends. But 
>> > a few users over internet claim support for NFS based storage from 
>> > Netapp in Prometheus. Just checking for opinions and feedback if this 
>> > storage backend is compatible enough for regular Prometheus operations.
>> >
>> NFS isn't a supported option for Prometheus. Prometheus can do a lot of 
>> I/O and therefore network storage often doesn't work all that well. You 
>> can also have issues if the filesystem isn't fully POSIX compliant.
>>
>> Just because it isn't supported doesn't mean it won't work. However 
>> there are issues that people find, and if you do find those the 
>> recommendation would always be to switch the filesystem to something 
>> local. So basically you'd be fairly on your own if you did have problems.
>>
>> -- 
>> Stuart Clark
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4799f7e1-376c-4772-8ab3-8abfaa0e0d21n%40googlegroups.com.

Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[tejaswini vadlamudi]

Thanks Stuart, now I understand support in Prometheus project.
Though renowned storage vendors claim full compliance and seem to work with 
Prometheus data, it is better to stay away from it.

Cheers, Teja
On Tuesday, May 3, 2022 at 9:53:52 PM UTC+2 Stuart Clark wrote:

> On 03/05/2022 01:34, tejaswini vadlamudi wrote:
> > Prometheus documentation recommends to avoid NFS storage backends. But 
> > a few users over internet claim support for NFS based storage from 
> > Netapp in Prometheus. Just checking for opinions and feedback if this 
> > storage backend is compatible enough for regular Prometheus operations.
> >
> NFS isn't a supported option for Prometheus. Prometheus can do a lot of 
> I/O and therefore network storage often doesn't work all that well. You 
> can also have issues if the filesystem isn't fully POSIX compliant.
>
> Just because it isn't supported doesn't mean it won't work. However 
> there are issues that people find, and if you do find those the 
> recommendation would always be to switch the filesystem to something 
> local. So basically you'd be fairly on your own if you did have problems.
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c436add2-e58b-43bb-99bd-bba0f11a4c23n%40googlegroups.com.

Re: [prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[Stuart Clark]

On 03/05/2022 01:34, tejaswini vadlamudi wrote:
Prometheus documentation recommends to avoid NFS storage backends. But 
a few users over internet claim support for NFS based storage from 
Netapp in Prometheus. Just checking for opinions and feedback if this 
storage backend is compatible enough for regular Prometheus operations.


NFS isn't a supported option for Prometheus. Prometheus can do a lot of 
I/O and therefore network storage often doesn't work all that well. You 
can also have issues if the filesystem isn't fully POSIX compliant.


Just because it isn't supported doesn't mean it won't work. However 
there are issues that people find, and if you do find those the 
recommendation would always be to switch the filesystem to something 
local. So basically you'd be fairly on your own if you did have problems.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/91e130f1-1a94-b31e-59e6-960aa2dfae37%40Jahingo.com.

[prometheus-users] Does Prometheus support Netapp Trident NFS storage backend?

[tejaswini vadlamudi]

Prometheus documentation recommends to avoid NFS storage backends. But a 
few users over internet claim support for NFS based storage from Netapp in 
Prometheus. Just checking for opinions and feedback if this storage backend 
is compatible enough for regular Prometheus operations.

Thanks,
Teja

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8f0e0281-da4e-40e9-9a83-32e6da699bf2n%40googlegroups.com.