I am not a developer of prometheus, just a satisfied user who sometimes shares some PromQL tips.
I do know that Ben Kochie is a major figure in the project, therefore I suggest you follow his advice, and the documented advice, which I would summarise as: "Don't do that". It is of course open source, and you are free to do what you like with it, but if you don't use it in the recommended way then you support yourself. I think that's fair. On Sunday, 2 February 2025 at 23:23:28 UTC Gavine Xue wrote: > Hi Brian, > > Hope you had a good weekend! > > I’d like to confirm the exact behavior when rolling back from *Prometheus > v3.x* (where *FormatV3 TSDB* is generated) to a *pre-v2.55 version*, such > as *v2.54.1*. > > The *v3 migration guide* states: > > "... you will only be able to downgrade to v2.55, not lower, without > losing your TSDB persistent data." > > This gives me the impression that *downgrading to v2.54.1 is still > possible*, but I’d appreciate clarification on the expected behavior: > > 1. *Would Prometheus v2.54.1 start after rollback, ignoring / losing > FormatV3 TSDB data, while FormatV2 blocks (if present) remain intact?* > 2. *Or is downgrading to v2.54.1 entirely unsupported, e.g. Prometheus > v2.54.1 won't start at all?* > > From reviewing the source code, this block: > 🔗 *db.go#L1600-L1615 > <https://github.com/prometheus/prometheus/blob/main/tsdb/db.go#L1600-L1615>* > suggests that *Prometheus may fail even though openBlocks() returns OK > with FormatV3 blocks put into corrupted*. Could you confirm if this is > the intended behavior? > Thanks! > > Best regards > Gavine > On Tuesday, 28 January 2025 at 11:23:12 am UTC+11 Jian Xue wrote: > >> Hey Brian, >> >> Thanks for the links — they are exactly what we’ve been referring to. >> >> It’s great to hear that 'in principle, it could still be compatible as >> far back as Prometheus 2.15.' If I’m interpreting this correctly, our >> initial thought should work, right? Specifically, simply changing the >> version from 3 to 2 in the index file for a V3-formatted TSDB block should >> make it recognizable and properly handled by Prometheus versions prior to >> 2.55, going back to 2.15. Of course, this is theoretical, and proper >> testing will be necessary once Prometheus releases a version that generates >> V3-formatted TSDB blocks. >> >> Bumping the version is absolutely justified given the breaking changes in >> the format. We are, however, seeking a feasible solution to meet our >> requirements — enabling rollback from Prometheus v3 to a version prior to >> v2.55 (though not as old as 2.15). >> >> Thanks! >> Best regards >> Gavine >> >> On Tue, 28 Jan 2025 at 00:20, 'Brian Candler' via Prometheus Users < >> [email protected]> wrote: >> >>> The release notes for v2.55 describe this as an "upcoming index v3", and >>> links to >>> https://github.com/prometheus/prometheus/pull/14934 >>> which in turn links to >>> https://github.com/prometheus/prometheus/issues/14749 >>> >>> That has the full details of what's compatible. If I read this >>> correctly, the proposal is to remove some obsolete fields. Once the format >>> change has been made (and it hasn't yet), in principle it could still be >>> compatible as far back as prometheus 2.15. >>> >>> However, not only prometheus reads these chunk files (there's also >>> thanos, mimir, ...) so good engineering practice says that along with >>> changing the format, the version number will be bumped. >>> >>> On Friday, 24 January 2025 at 13:24:33 UTC Gavine Xue wrote: >>> >>>> Hi Brian, >>>> >>>> Thanks for your prompt reply. >>>> >>>> We were thinking the same — 'it's fairly obvious that won't work.' >>>> However, when we quickly reviewed the changes in v2.55.0, we noticed that >>>> it primarily makes FormatV3 recognizable while applying the same logic as >>>> FormatV2 (Backward compatibility with upcoming index v3 · >>>> prometheus/prometheus@5ccb069 · GitHub >>>> <https://github.com/prometheus/prometheus/commit/5ccb0694146d666e49068e82c159300fedef6b76> >>>> ). >>>> Other changes under the tsdb folder seem irrelevant (though we could be >>>> mistaken, as we didn’t dive into the details). >>>> >>>> Additionally, Prometheus V2, starting from some versions, has already >>>> stopped reading/parsing Label Index 1...N and the Label Index Table — both >>>> of which are removed in Format V3. All this makes us wonder if simply >>>> changing the version number would suffice. >>>> >>>> Are there any other changes in v2.55.0 that we should inspect further? >>>> >>>> Another question: Have we decided which v3 version will make this >>>> change effective? The current v3 is still generating FormatV2 TSDB. >>>> Thanks! >>>> >>>> Best regards >>>> Gavine >>>> >>>> On Friday, 24 January 2025 at 7:49:46 pm UTC+11 Brian Candler wrote: >>>> >>>>> On Monday, 20 January 2025 at 13:32:11 UTC Harpreet Rekhi wrote: >>>>> >>>>> Two steps rollback is also not supported ? Like rollback from v3.x to >>>>> v2.55 and then rollback from v2.55 to v2.53 without breaking the storage >>>>> format or storage compatibility? >>>>> >>>>> >>>>> That is exactly what Ben said. v2.55 knows how to work with the data >>>>> format from v3, but will not convert it back to the older format. >>>>> >>>>> It sounds to me like you would be best served by staying on the LTS >>>>> version v2.53, which still has bug and security fix support. Why do you >>>>> want to run anything later? >>>>> >>>>> If you *absolutely* need new features introduced in later versions, >>>>> then you will have to remain between versions v2.55 and v3.x - as there >>>>> is >>>>> no rollback to anything earlier than v2.55 without losing data. But >>>>> that's >>>>> fine, because if you *need* those new features then you wouldn't be able >>>>> to >>>>> rollback anyway. Bear in mind that you will also have to keep upgrading >>>>> to >>>>> all v3.x versions to get bugfixes and security fixes, until the next LTS >>>>> release comes along. >>>>> >>>>> If you just want to test the newer versions then do that on a >>>>> non-production system where you know you'll never need to roll back prior >>>>> to v2.55 >>>>> >>>>> Gavine Xue wrote: >>>>> > do you foresee any issues if we develop a conversion tool to change >>>>> the version number in index files back to 2 before a rollback? >>>>> >>>>> I think it's fairly obvious that won't work. If the *only* difference >>>>> between the data formats was the version number, then there would have >>>>> been >>>>> no need for a change in the version number. The change in the version >>>>> number implies that the way that the data is stored has been changed in >>>>> an >>>>> incompatible way. >>>>> >>>> -- >>> >> You received this message because you are subscribed to the Google Groups >>> "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion visit >>> https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/154d512d-cc68-4273-9f65-c35d5a834fe7n%40googlegroups.com.
