I think your problem has been solved in Prometheus v2.53.5: [ENHANCEMENT] TSDB: Add backward compatibility with the upcoming TSDB block index v3 #16762 <https://github.com/prometheus/prometheus/pull/16762>
On Tuesday, 4 February 2025 at 01:10:46 UTC Gavine Xue wrote: > Thanks, Brian, for the heads-up. > > Hi Ben, > > Could you please help clarify the following questions? > > 1. > > Regarding the statement in the V3 migration guide: *"... you will only > be able to downgrade to v2.55, not lower, without losing your TSDB > persistent data." *— does this mean that downgrading from v3.x to a > pre-v2.55.0 version is completely unsupported, or is it technically > possible but would result in the loss of FormatV3 data after rollback? > This > distinction is crucial for our customers. > 2. > > Is there a version — e.g., in a dev branch — where this change is > already effective (i.e., generating FormatV3 TSDB data) so that we can > test > it? > > Thanks! > > Best regards > Gavine > > On Monday, 3 February 2025 at 7:56:41 pm UTC+11 Brian Candler wrote: > >> I am not a developer of prometheus, just a satisfied user who sometimes >> shares some PromQL tips. >> >> I do know that Ben Kochie is a major figure in the project, therefore I >> suggest you follow his advice, and the documented advice, which I would >> summarise as: "Don't do that". >> >> It is of course open source, and you are free to do what you like with >> it, but if you don't use it in the recommended way then you support >> yourself. I think that's fair. >> >> On Sunday, 2 February 2025 at 23:23:28 UTC Gavine Xue wrote: >> >>> Hi Brian, >>> >>> Hope you had a good weekend! >>> >>> I’d like to confirm the exact behavior when rolling back from *Prometheus >>> v3.x* (where *FormatV3 TSDB* is generated) to a *pre-v2.55 version*, >>> such as *v2.54.1*. >>> >>> The *v3 migration guide* states: >>> >>> "... you will only be able to downgrade to v2.55, not lower, without >>> losing your TSDB persistent data." >>> >>> This gives me the impression that *downgrading to v2.54.1 is still >>> possible*, but I’d appreciate clarification on the expected behavior: >>> >>> 1. *Would Prometheus v2.54.1 start after rollback, ignoring / losing >>> FormatV3 TSDB data, while FormatV2 blocks (if present) remain intact?* >>> 2. *Or is downgrading to v2.54.1 entirely unsupported, e.g. >>> Prometheus v2.54.1 won't start at all?* >>> >>> From reviewing the source code, this block: >>> 🔗 *db.go#L1600-L1615 >>> <https://github.com/prometheus/prometheus/blob/main/tsdb/db.go#L1600-L1615>* >>> suggests that *Prometheus may fail even though openBlocks() returns OK >>> with FormatV3 blocks put into corrupted*. Could you confirm if this is >>> the intended behavior? >>> Thanks! >>> >>> Best regards >>> Gavine >>> On Tuesday, 28 January 2025 at 11:23:12 am UTC+11 Jian Xue wrote: >>> >>>> Hey Brian, >>>> >>>> Thanks for the links — they are exactly what we’ve been referring to. >>>> >>>> It’s great to hear that 'in principle, it could still be compatible as >>>> far back as Prometheus 2.15.' If I’m interpreting this correctly, our >>>> initial thought should work, right? Specifically, simply changing the >>>> version from 3 to 2 in the index file for a V3-formatted TSDB block should >>>> make it recognizable and properly handled by Prometheus versions prior to >>>> 2.55, going back to 2.15. Of course, this is theoretical, and proper >>>> testing will be necessary once Prometheus releases a version that >>>> generates >>>> V3-formatted TSDB blocks. >>>> >>>> Bumping the version is absolutely justified given the breaking changes >>>> in the format. We are, however, seeking a feasible solution to meet our >>>> requirements — enabling rollback from Prometheus v3 to a version prior to >>>> v2.55 (though not as old as 2.15). >>>> >>>> Thanks! >>>> Best regards >>>> Gavine >>>> >>>> On Tue, 28 Jan 2025 at 00:20, 'Brian Candler' via Prometheus Users < >>>> [email protected]> wrote: >>>> >>>>> The release notes for v2.55 describe this as an "upcoming index v3", >>>>> and links to >>>>> https://github.com/prometheus/prometheus/pull/14934 >>>>> which in turn links to >>>>> https://github.com/prometheus/prometheus/issues/14749 >>>>> >>>>> That has the full details of what's compatible. If I read this >>>>> correctly, the proposal is to remove some obsolete fields. Once the >>>>> format >>>>> change has been made (and it hasn't yet), in principle it could still be >>>>> compatible as far back as prometheus 2.15. >>>>> >>>>> However, not only prometheus reads these chunk files (there's also >>>>> thanos, mimir, ...) so good engineering practice says that along with >>>>> changing the format, the version number will be bumped. >>>>> >>>>> On Friday, 24 January 2025 at 13:24:33 UTC Gavine Xue wrote: >>>>> >>>>>> Hi Brian, >>>>>> >>>>>> Thanks for your prompt reply. >>>>>> >>>>>> We were thinking the same — 'it's fairly obvious that won't work.' >>>>>> However, when we quickly reviewed the changes in v2.55.0, we noticed >>>>>> that >>>>>> it primarily makes FormatV3 recognizable while applying the same logic >>>>>> as >>>>>> FormatV2 (Backward compatibility with upcoming index v3 · >>>>>> prometheus/prometheus@5ccb069 · GitHub >>>>>> <https://github.com/prometheus/prometheus/commit/5ccb0694146d666e49068e82c159300fedef6b76> >>>>>> ). >>>>>> Other changes under the tsdb folder seem irrelevant (though we could be >>>>>> mistaken, as we didn’t dive into the details). >>>>>> >>>>>> Additionally, Prometheus V2, starting from some versions, has already >>>>>> stopped reading/parsing Label Index 1...N and the Label Index Table — >>>>>> both >>>>>> of which are removed in Format V3. All this makes us wonder if simply >>>>>> changing the version number would suffice. >>>>>> >>>>>> Are there any other changes in v2.55.0 that we should inspect further? >>>>>> >>>>>> Another question: Have we decided which v3 version will make this >>>>>> change effective? The current v3 is still generating FormatV2 TSDB. >>>>>> Thanks! >>>>>> >>>>>> Best regards >>>>>> Gavine >>>>>> >>>>>> On Friday, 24 January 2025 at 7:49:46 pm UTC+11 Brian Candler wrote: >>>>>> >>>>>>> On Monday, 20 January 2025 at 13:32:11 UTC Harpreet Rekhi wrote: >>>>>>> >>>>>>> Two steps rollback is also not supported ? Like rollback from v3.x >>>>>>> to v2.55 and then rollback from v2.55 to v2.53 without breaking the >>>>>>> storage >>>>>>> format or storage compatibility? >>>>>>> >>>>>>> >>>>>>> That is exactly what Ben said. v2.55 knows how to work with the data >>>>>>> format from v3, but will not convert it back to the older format. >>>>>>> >>>>>>> It sounds to me like you would be best served by staying on the LTS >>>>>>> version v2.53, which still has bug and security fix support. Why do >>>>>>> you >>>>>>> want to run anything later? >>>>>>> >>>>>>> If you *absolutely* need new features introduced in later versions, >>>>>>> then you will have to remain between versions v2.55 and v3.x - as there >>>>>>> is >>>>>>> no rollback to anything earlier than v2.55 without losing data. But >>>>>>> that's >>>>>>> fine, because if you *need* those new features then you wouldn't be >>>>>>> able to >>>>>>> rollback anyway. Bear in mind that you will also have to keep >>>>>>> upgrading to >>>>>>> all v3.x versions to get bugfixes and security fixes, until the next >>>>>>> LTS >>>>>>> release comes along. >>>>>>> >>>>>>> If you just want to test the newer versions then do that on a >>>>>>> non-production system where you know you'll never need to roll back >>>>>>> prior >>>>>>> to v2.55 >>>>>>> >>>>>>> Gavine Xue wrote: >>>>>>> > do you foresee any issues if we develop a conversion tool to >>>>>>> change the version number in index files back to 2 before a rollback? >>>>>>> >>>>>>> I think it's fairly obvious that won't work. If the *only* >>>>>>> difference between the data formats was the version number, then there >>>>>>> would have been no need for a change in the version number. The change >>>>>>> in >>>>>>> the version number implies that the way that the data is stored has >>>>>>> been >>>>>>> changed in an incompatible way. >>>>>>> >>>>>> -- >>>>> >>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion visit >>>>> https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/9b1fddc4-a260-479a-be68-4ab2e1aa6537n%40googlegroups.com.
