Hi Brian, Hope you had a good weekend!
I’d like to confirm the exact behavior when rolling back from *Prometheus v3.x* (where *FormatV3 TSDB* is generated) to a *pre-v2.55 version*, such as *v2.54.1*. The *v3 migration guide* states: "... you will only be able to downgrade to v2.55, not lower, without losing your TSDB persistent data." This gives me the impression that *downgrading to v2.54.1 is still possible*, but I’d appreciate clarification on the expected behavior: 1. *Would Prometheus v2.54.1 start after rollback, ignoring / losing FormatV3 TSDB data, while FormatV2 blocks (if present) remain intact?* 2. *Or is downgrading to v2.54.1 entirely unsupported, e.g. Prometheus v2.54.1 won't start at all?* >From reviewing the source code, this block: 🔗 *db.go#L1600-L1615 <https://github.com/prometheus/prometheus/blob/main/tsdb/db.go#L1600-L1615>* suggests that *Prometheus may fail even though openBlocks() returns OK with FormatV3 blocks put into corrupted*. Could you confirm if this is the intended behavior? Thanks! Best regards Gavine On Tuesday, 28 January 2025 at 11:23:12 am UTC+11 Jian Xue wrote: > Hey Brian, > > Thanks for the links — they are exactly what we’ve been referring to. > > It’s great to hear that 'in principle, it could still be compatible as far > back as Prometheus 2.15.' If I’m interpreting this correctly, our initial > thought should work, right? Specifically, simply changing the version from > 3 to 2 in the index file for a V3-formatted TSDB block should make it > recognizable and properly handled by Prometheus versions prior to 2.55, > going back to 2.15. Of course, this is theoretical, and proper testing > will be necessary once Prometheus releases a version that generates > V3-formatted TSDB blocks. > > Bumping the version is absolutely justified given the breaking changes in > the format. We are, however, seeking a feasible solution to meet our > requirements — enabling rollback from Prometheus v3 to a version prior to > v2.55 (though not as old as 2.15). > > Thanks! > Best regards > Gavine > > On Tue, 28 Jan 2025 at 00:20, 'Brian Candler' via Prometheus Users < > [email protected]> wrote: > >> The release notes for v2.55 describe this as an "upcoming index v3", and >> links to >> https://github.com/prometheus/prometheus/pull/14934 >> which in turn links to >> https://github.com/prometheus/prometheus/issues/14749 >> >> That has the full details of what's compatible. If I read this correctly, >> the proposal is to remove some obsolete fields. Once the format change has >> been made (and it hasn't yet), in principle it could still be compatible as >> far back as prometheus 2.15. >> >> However, not only prometheus reads these chunk files (there's also >> thanos, mimir, ...) so good engineering practice says that along with >> changing the format, the version number will be bumped. >> >> On Friday, 24 January 2025 at 13:24:33 UTC Gavine Xue wrote: >> >>> Hi Brian, >>> >>> Thanks for your prompt reply. >>> >>> We were thinking the same — 'it's fairly obvious that won't work.' >>> However, when we quickly reviewed the changes in v2.55.0, we noticed that >>> it primarily makes FormatV3 recognizable while applying the same logic as >>> FormatV2 (Backward compatibility with upcoming index v3 · >>> prometheus/prometheus@5ccb069 · GitHub >>> <https://github.com/prometheus/prometheus/commit/5ccb0694146d666e49068e82c159300fedef6b76> >>> ). >>> Other changes under the tsdb folder seem irrelevant (though we could be >>> mistaken, as we didn’t dive into the details). >>> >>> Additionally, Prometheus V2, starting from some versions, has already >>> stopped reading/parsing Label Index 1...N and the Label Index Table — both >>> of which are removed in Format V3. All this makes us wonder if simply >>> changing the version number would suffice. >>> >>> Are there any other changes in v2.55.0 that we should inspect further? >>> >>> Another question: Have we decided which v3 version will make this change >>> effective? The current v3 is still generating FormatV2 TSDB. >>> Thanks! >>> >>> Best regards >>> Gavine >>> >>> On Friday, 24 January 2025 at 7:49:46 pm UTC+11 Brian Candler wrote: >>> >>>> On Monday, 20 January 2025 at 13:32:11 UTC Harpreet Rekhi wrote: >>>> >>>> Two steps rollback is also not supported ? Like rollback from v3.x to >>>> v2.55 and then rollback from v2.55 to v2.53 without breaking the storage >>>> format or storage compatibility? >>>> >>>> >>>> That is exactly what Ben said. v2.55 knows how to work with the data >>>> format from v3, but will not convert it back to the older format. >>>> >>>> It sounds to me like you would be best served by staying on the LTS >>>> version v2.53, which still has bug and security fix support. Why do you >>>> want to run anything later? >>>> >>>> If you *absolutely* need new features introduced in later versions, >>>> then you will have to remain between versions v2.55 and v3.x - as there is >>>> no rollback to anything earlier than v2.55 without losing data. But that's >>>> fine, because if you *need* those new features then you wouldn't be able >>>> to >>>> rollback anyway. Bear in mind that you will also have to keep upgrading >>>> to >>>> all v3.x versions to get bugfixes and security fixes, until the next LTS >>>> release comes along. >>>> >>>> If you just want to test the newer versions then do that on a >>>> non-production system where you know you'll never need to roll back prior >>>> to v2.55 >>>> >>>> Gavine Xue wrote: >>>> > do you foresee any issues if we develop a conversion tool to change >>>> the version number in index files back to 2 before a rollback? >>>> >>>> I think it's fairly obvious that won't work. If the *only* difference >>>> between the data formats was the version number, then there would have >>>> been >>>> no need for a change in the version number. The change in the version >>>> number implies that the way that the data is stored has been changed in an >>>> incompatible way. >>>> >>> -- >> > You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion visit >> https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/aad36298-33ae-4291-9ec2-c4d0a6782efan%40googlegroups.com.
