Thanks, Brian, for the heads-up. Hi Ben,
Could you please help clarify the following questions? 1. Regarding the statement in the V3 migration guide: *"... you will only be able to downgrade to v2.55, not lower, without losing your TSDB persistent data." *— does this mean that downgrading from v3.x to a pre-v2.55.0 version is completely unsupported, or is it technically possible but would result in the loss of FormatV3 data after rollback? This distinction is crucial for our customers. 2. Is there a version — e.g., in a dev branch — where this change is already effective (i.e., generating FormatV3 TSDB data) so that we can test it? Thanks! Best regards Gavine On Monday, 3 February 2025 at 7:56:41 pm UTC+11 Brian Candler wrote: > I am not a developer of prometheus, just a satisfied user who sometimes > shares some PromQL tips. > > I do know that Ben Kochie is a major figure in the project, therefore I > suggest you follow his advice, and the documented advice, which I would > summarise as: "Don't do that". > > It is of course open source, and you are free to do what you like with it, > but if you don't use it in the recommended way then you support yourself. I > think that's fair. > > On Sunday, 2 February 2025 at 23:23:28 UTC Gavine Xue wrote: > >> Hi Brian, >> >> Hope you had a good weekend! >> >> I’d like to confirm the exact behavior when rolling back from *Prometheus >> v3.x* (where *FormatV3 TSDB* is generated) to a *pre-v2.55 version*, >> such as *v2.54.1*. >> >> The *v3 migration guide* states: >> >> "... you will only be able to downgrade to v2.55, not lower, without >> losing your TSDB persistent data." >> >> This gives me the impression that *downgrading to v2.54.1 is still >> possible*, but I’d appreciate clarification on the expected behavior: >> >> 1. *Would Prometheus v2.54.1 start after rollback, ignoring / losing >> FormatV3 TSDB data, while FormatV2 blocks (if present) remain intact?* >> 2. *Or is downgrading to v2.54.1 entirely unsupported, e.g. >> Prometheus v2.54.1 won't start at all?* >> >> From reviewing the source code, this block: >> 🔗 *db.go#L1600-L1615 >> <https://github.com/prometheus/prometheus/blob/main/tsdb/db.go#L1600-L1615>* >> suggests that *Prometheus may fail even though openBlocks() returns OK >> with FormatV3 blocks put into corrupted*. Could you confirm if this is >> the intended behavior? >> Thanks! >> >> Best regards >> Gavine >> On Tuesday, 28 January 2025 at 11:23:12 am UTC+11 Jian Xue wrote: >> >>> Hey Brian, >>> >>> Thanks for the links — they are exactly what we’ve been referring to. >>> >>> It’s great to hear that 'in principle, it could still be compatible as >>> far back as Prometheus 2.15.' If I’m interpreting this correctly, our >>> initial thought should work, right? Specifically, simply changing the >>> version from 3 to 2 in the index file for a V3-formatted TSDB block should >>> make it recognizable and properly handled by Prometheus versions prior to >>> 2.55, going back to 2.15. Of course, this is theoretical, and proper >>> testing will be necessary once Prometheus releases a version that generates >>> V3-formatted TSDB blocks. >>> >>> Bumping the version is absolutely justified given the breaking changes >>> in the format. We are, however, seeking a feasible solution to meet our >>> requirements — enabling rollback from Prometheus v3 to a version prior to >>> v2.55 (though not as old as 2.15). >>> >>> Thanks! >>> Best regards >>> Gavine >>> >>> On Tue, 28 Jan 2025 at 00:20, 'Brian Candler' via Prometheus Users < >>> [email protected]> wrote: >>> >>>> The release notes for v2.55 describe this as an "upcoming index v3", >>>> and links to >>>> https://github.com/prometheus/prometheus/pull/14934 >>>> which in turn links to >>>> https://github.com/prometheus/prometheus/issues/14749 >>>> >>>> That has the full details of what's compatible. If I read this >>>> correctly, the proposal is to remove some obsolete fields. Once the format >>>> change has been made (and it hasn't yet), in principle it could still be >>>> compatible as far back as prometheus 2.15. >>>> >>>> However, not only prometheus reads these chunk files (there's also >>>> thanos, mimir, ...) so good engineering practice says that along with >>>> changing the format, the version number will be bumped. >>>> >>>> On Friday, 24 January 2025 at 13:24:33 UTC Gavine Xue wrote: >>>> >>>>> Hi Brian, >>>>> >>>>> Thanks for your prompt reply. >>>>> >>>>> We were thinking the same — 'it's fairly obvious that won't work.' >>>>> However, when we quickly reviewed the changes in v2.55.0, we noticed that >>>>> it primarily makes FormatV3 recognizable while applying the same logic as >>>>> FormatV2 (Backward compatibility with upcoming index v3 · >>>>> prometheus/prometheus@5ccb069 · GitHub >>>>> <https://github.com/prometheus/prometheus/commit/5ccb0694146d666e49068e82c159300fedef6b76> >>>>> ). >>>>> Other changes under the tsdb folder seem irrelevant (though we could be >>>>> mistaken, as we didn’t dive into the details). >>>>> >>>>> Additionally, Prometheus V2, starting from some versions, has already >>>>> stopped reading/parsing Label Index 1...N and the Label Index Table — >>>>> both >>>>> of which are removed in Format V3. All this makes us wonder if simply >>>>> changing the version number would suffice. >>>>> >>>>> Are there any other changes in v2.55.0 that we should inspect further? >>>>> >>>>> Another question: Have we decided which v3 version will make this >>>>> change effective? The current v3 is still generating FormatV2 TSDB. >>>>> Thanks! >>>>> >>>>> Best regards >>>>> Gavine >>>>> >>>>> On Friday, 24 January 2025 at 7:49:46 pm UTC+11 Brian Candler wrote: >>>>> >>>>>> On Monday, 20 January 2025 at 13:32:11 UTC Harpreet Rekhi wrote: >>>>>> >>>>>> Two steps rollback is also not supported ? Like rollback from v3.x to >>>>>> v2.55 and then rollback from v2.55 to v2.53 without breaking the storage >>>>>> format or storage compatibility? >>>>>> >>>>>> >>>>>> That is exactly what Ben said. v2.55 knows how to work with the data >>>>>> format from v3, but will not convert it back to the older format. >>>>>> >>>>>> It sounds to me like you would be best served by staying on the LTS >>>>>> version v2.53, which still has bug and security fix support. Why do you >>>>>> want to run anything later? >>>>>> >>>>>> If you *absolutely* need new features introduced in later versions, >>>>>> then you will have to remain between versions v2.55 and v3.x - as there >>>>>> is >>>>>> no rollback to anything earlier than v2.55 without losing data. But >>>>>> that's >>>>>> fine, because if you *need* those new features then you wouldn't be able >>>>>> to >>>>>> rollback anyway. Bear in mind that you will also have to keep upgrading >>>>>> to >>>>>> all v3.x versions to get bugfixes and security fixes, until the next LTS >>>>>> release comes along. >>>>>> >>>>>> If you just want to test the newer versions then do that on a >>>>>> non-production system where you know you'll never need to roll back >>>>>> prior >>>>>> to v2.55 >>>>>> >>>>>> Gavine Xue wrote: >>>>>> > do you foresee any issues if we develop a conversion tool to >>>>>> change the version number in index files back to 2 before a rollback? >>>>>> >>>>>> I think it's fairly obvious that won't work. If the *only* difference >>>>>> between the data formats was the version number, then there would have >>>>>> been >>>>>> no need for a change in the version number. The change in the version >>>>>> number implies that the way that the data is stored has been changed in >>>>>> an >>>>>> incompatible way. >>>>>> >>>>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion visit >>>> https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/1fa127fb-9c9f-4018-9859-18106cb8354bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/0e158f50-c701-4124-b1ba-682e3c87ee6cn%40googlegroups.com.
