Re: [prometheus-users] Re: Data getting corrpted - missing meta.json

2020-04-03 Thread Robin Pharaoh
Hi Christian, This does appear to be the issue. In the Azure ecosystem Azure Disks should be posix compliant, would switching to those resolve the issue or would we still have issues with NFS? Reference at the very bottom of this link: https://social.msdn.microsoft.com/Forums/en-US/0e19f5f8-176

Re: [prometheus-users] Re: Data getting corrpted - missing meta.json

2020-04-03 Thread Christian Hoffmann
Hi Robin, On 4/3/20 8:36 PM, Robin Pharaoh wrote: > Was this ever resolved? We are hitting a the exact same issue right now. > > We have a single instance of prometheus > We are using Azure File Share with a volume claim I don't have any Azure knowledge, but I assume this is a SMB-based mount? T

[prometheus-users] Re: Data getting corrpted - missing meta.json

2020-04-03 Thread Robin Pharaoh
Was this ever resolved? We are hitting a the exact same issue right now. We have a single instance of prometheus We are using Azure File Share with a volume claim On Tuesday, February 18, 2020 at 10:16:44 PM UTC-5, Guru SD wrote: > > Hello, > > We have a clustered Prometheus setup but with a comm

Re: [prometheus-users] Re: Data getting corrpted - missing meta.json

2020-02-20 Thread Guru SD
Thanks for confirming, kind of thought this was the problem. Can you pls help with which file system to use? We are running Ubuntu on Azure VMs. Is it ext4? On Thursday, 20 February 2020 18:36:45 UTC+5:30, Stuart Clark wrote: > > If you have more than one copy of Prometheus running against the

Re: [prometheus-users] Re: Data getting corrpted - missing meta.json

2020-02-20 Thread Stuart Clark
If you have more than one copy of Prometheus running against the same data directory you will cause corruption. They need to be using separate directories and the use of NFS is generally not recommended. On 20 February 2020 12:40:59 GMT, Guru SD wrote: >Yes we have two Prometheus servers acce

[prometheus-users] Re: Data getting corrpted - missing meta.json

2020-02-20 Thread Guru SD
Yes we have two Prometheus servers accessing the same storage backend. Not sure if it is via NFS. They have configured to use the same storage.tsdb.path on one of the two servers. The servers are Azure VMs running on Ubuntu. Using the latest version of Prometheus. On Wednesday, 19 February 202

[prometheus-users] Re: Data getting corrpted - missing meta.json

2020-02-19 Thread Brian Candler
On Wednesday, 19 February 2020 03:16:44 UTC, Guru SD wrote: > > We have a clustered Prometheus setup but with a common storage disk. > Please can you be more specific about what you mean by "common storage disk" and how it is configured. Do you have multiple prometheus servers accessing the same