> It is not really possible to make the shims work the same way because there's not enough information available to determine whether an adjustment needs to be made.
But since you're shimming pytz, don't you, by definition, have access to the all the same information that it has? So, for example, you could wrap pytz and keep a shadow copy of the pytz tzobject inside your tzobject, and use that to determine the correct behavior whenever a pytz-specific call is made. So when localize() is called you call localize on your internal object, and store that pytz EST tzobject. Then when normalize() is called you use that to get pytz's version of the EDT time. > This occurs because localized pytz zones are different tzinfo objects, and as such comparisons and subtraction use inter-zone semantics. Thank you for that example, I hadn't considered that. Unfortunately that is another fundamental incompatibility between pytz and any shim-around-zoneinfo (here <https://repl.it/@severian/pytzshim-subtract> is a runnable version). I can't think of any way around that one. > Of course, there is another option, which is to, rather than adopting a wrapper around zoneinfo, adopt a wrapper around pytz that does not follow PEP 495, but instead just deprecates `pytz`'s API and tells people to turn on the "use zoneinfo" feature flag. Agreed, that is the other main option. It has a few advantages: - It's backwards-compatible. - Because it's backwards compatible it could be adopted in 3.2, allowing a complete transition to zoneinfo by 4.2, a full two years earlier than the shims approach. - It only requires users to think about the change once, when they opt in to the new approach. Using the shim means you have to think about this issue at least twice: once when the shim is dropped in and you have to figure out if you're affected by the backwards incompatibilities; and once (or more) when you actually make the change (or a series of changes) over to the native zoneinfo style. The main disadvantage—and a real one—is that it's more work for Django. Cheers, Kevin On Friday, October 9, 2020 at 8:06:49 AM UTC-7 Paul Ganssle wrote: > Before looking at alternatives, I wonder if we can just change the shims > package to make it fully backwards compatible? Right now the shims version > of normalize() > <https://github.com/pganssle/pytz-deprecation-shim/blob/47bd4bdd9346cafa6c6d66817082ccce099890ad/src/pytz_deprecation_shim/_impl.py#L265> > > is essentially a noop. Paul, couldn't it actually attempt to adjust the > time the way pytz does? Perhaps by wrapping pytz itself, and calling its > normalize() from the corresponding pytz timezone; or by simply replicating > its time-changing logic? Apologies if that's a naive question. > > > It is not really possible to make the shims work the same way because > there's not enough information available to determine whether an adjustment > needs to be made. The reason that `normalize` works is that pytz attaches > different `tzinfo` objects representing fixed offsets (with a reference to > the time zone they represent) to the datetime. If arithmetic creates an > invalid datetime (i.e. a datetime in mid-June 2020 with EST attached), > `normalize` corrects this by attaching a `tzinfo` representing the correct > offset — and it does that by assuming that the UTC datetime represented by > the erroneous fixed offset is correct. With PEP 495-style zones, you never > create those datetimes with erroneous offsets, so there's no way to tell > whether a correction is required. > > For example: > >>> from datetime import datetime, timedelta > >>> from zoneinfo import ZoneInfo > > >>> NYC = ZoneInfo("America/New_York") > >>> dt0 = datetime(2020, 1, 1, tzinfo=NYC) > >>> dt1 = datetime(2020, 7, 1, tzinfo=NYC) > > >>> print(dt0) > 2020-01-01 00:00:00-05:00 > >>> print(dt1) > 2020-07-01 00:00:00-04:00 > > >>> print(dt0 + timedelta(days=183)) > 2020-07-02 00:00:00-04:00 > >>> print(dt1 + timedelta(days=1)) > 2020-07-02 00:00:00-04:00 > > Note that the two endpoints are identical, despite the fact that one of > them spans a DST transition and the other one doesn't. Since the input to > `normalize` is just a datetime and it's assumed that this path-dependence > would show up as an inconsistency in the offset, there's nothing we can do > here other than to actually have all the same problems as pytz. > > Of course, there is another option, which is to, rather than adopting a > wrapper around zoneinfo, adopt a wrapper around pytz that does *not* > follow PEP 495, but instead just deprecates `pytz`'s API and tells people > to turn on the "use zoneinfo" feature flag. It has the upside of being > fully backwards-compatible, but the downside of prolonging dependence on > pytz. > > Another option is to modify the shims so that `normalize` always raises an > exception instead of a warning (or maybe it raises an exception for > anything except UTC and fixed offsets). In that case, version 4.0 will > *mostly* just work and start raising deprecation warnings, but there will > be a hard breakage for anyone who would be negatively affected by the > change in semantics. This *would* still leave a possible problem in the > other direction, though: > > >>> from datetime import datetime, timedelta > >>> from zoneinfo import ZoneInfo > >>> import pytz > >>> NYC_p = pytz.timezone("America/New_York") > >>> NYC = ZoneInfo("America/New_York") > > >>> dtp_0 = NYC_p.localize(datetime(2020, 10, 31, 12)) > >>> dtp_1 = NYC_p.localize(datetime(2020, 11, 1, 12)) > >>> (dtp_1 - dtp_0 ) / timedelta(hours=1) > 25.0 > > >>> dtz_0 = datetime(2020, 10, 31, 12, tzinfo=NYC) > >>> dtz_1 = datetime(2020, 11, 1, 12, tzinfo=NYC) > >>> (dtz_1 - dtz_0) / timedelta(hours=1) > 24.0 > > This occurs because localized pytz zones are different tzinfo objects, and > as such comparisons and subtraction use inter-zone semantics. Of course, > you'll have this same problem even with a "hard break", since unlike > invocation of `normalize` and `localize`, subtraction operations will > succeed if you swap out the attached tzinfo for a zoneinfo tzinfo. > > If we go with any variation of using shim-around-zoneinfo like > pytz-deprecation-shim, I'd say those shims need to be introduced as a > breaking change in Django 4.0. If we go with shim-around-pytz, I think that > can safely be introduced in 3.2 (though that would *require* > simultaneously adding support for using zoneinfo, and even then it might > *mostly* force people to either do the migration in a single huge step or > to involve some wrapper functions for handling the period of time where the > time zone type is not consistent throughout the application). > > Best, > Paul > On 10/9/20 9:31 AM, Kevin Henry wrote: > > I think that the simplest approach—the one that would result in the least > amount of total work for both Django and its users—would be to adopt Nick's > suggestion and just switch to zoneinfo in 4.0. The problem is that it's > very hard to square that with Django's stability policy: "We’ll only break > backwards compatibility of these APIs without a deprecation process if a > bug or security hole makes it completely unavoidable." > > If we're going to follow the deprecation process, then there needs to be > some overlap where both ways of doing things are possible. The shims > package is a promising approach, but the fact that it's not actually > backwards compatible with pytz is a serious problem. Adopting it directly > as Carlton proposes also seems to violate the stability policy, albeit in a > less severe way. > > > Kevin > > On Thursday, October 8, 2020 at 11:35:21 PM UTC-7 smi...@gmail.com wrote: > >> Hi All, >> >> While I understand the desire to have an early opt-in for some I think >> the important question here is the deprecation warnings. The recent URL() >> change showed that no matter how long there is a new way some?/many? folk >> won't change until they need to. >> >> Nick -- if we introduced a breaking change in 4.0, would that not have >> the same impact on folk upgrading to 4.2LTS from 3.2LTS as that which >> Carlton is concerned about (3.2 from 2.2), albeit a few years further into >> the future. >> >> >> David >> >> On Thursday, 8 October 2020 at 09:08:50 UTC+1 jure.er...@gmail.com wrote: >> >>> I would definitely be in favor of an opt-in: it would give developers >>> time to move to the new system at their convenience. >>> >>> Example: we're about to try and tackle the TZ issue in our apps and we >>> want to do it "globally" with one definitive solution. I'd much rather do >>> it on a library that is currently favoured, but not yet default than on a >>> deprecated one, even if it's not yet officially deprecated. We do have some >>> "import pytz", but currently they are few. Once we have a proper approach >>> to handling timezone stuff, there's likely going to be more of them... or >>> less, depending on the solution ;-) >>> >>> LP, >>> Jure >>> On 7. 10. 20 17:25, Paul Ganssle wrote: >>> >>> This sounds like a reasonable timeline to me. I think the breakage will >>> be relatively small because I suspect many end-users don't really even know >>> to use `normalize` in the first place, and when introducing the shim into a >>> fundamental library at work I did not get a huge number of breakages, but I >>> am still convinced that it is reasonably categorized as a breaking change. >>> >>> I do think that there's one additional stage that we need to add here >>> (and we chatted about this on twitter a bit), which is a stage that is >>> fully backwards compatible where Django supports using non-pytz zones for >>> users who bring their own time zone. I suspect that will help ease any >>> breaking pain between 3.2 and 4.0, because no one would be forced to make >>> any changes, but end users could proactively migrate to zoneinfo for a >>> smoother transition. >>> >>> I think most of what needs to be done is already in my original PR, it >>> just needs a little conditional logic to handle pytz as well as the shim. >>> >>> I am not sure how you feel about feature flags, but as a "nice to have", >>> I imagine it would also be possible to add a feature flag that opts you in >>> to `zoneinfo` as time zone provider even in 3.2, so that people can jump >>> straight to the 5.0 behavior if they are ready for it. >>> >>> I should be able to devote some time to at least the first part — making >>> Django compatible with zoneinfo even if not actively using it — but likely >>> not for a few weeks at minimum. If anyone wants to jump on either of these >>> ahead of me I don't mind at all and feel free to ping me for review. >>> >>> Best, >>> Paul >>> On 10/7/20 10:48 AM, Carlton Gibson wrote: >>> >>> Hi Paul. >>> >>> Thanks for the input here, and for your patience >>> >>> > I am fairly certain this is going to be a tricky migration and will >>> inevitably come with *some* user pain. I don't think this will be >>> Python 2 → 3 style pain, but some users who have been doing the "right >>> thing" with pytz will need to make changes to their code in the long run, >>> which is unfortunate. >>> >>> Looking at all the docs, your migration guide on pytz_deprecation_shim, >>> the example Kevin gave <https://repl.it/@severian/pytzshim#main.py>, >>> where we do some arithmetic in a local timezone, and call `normalize()` in >>> case we crossed a DST boundary, there's no way we can do this without >>> forcing a breaking change somewhere. >>> >>> So, probably, I've been staring at this too long today, but I think we >>> should introduce the shim in Django 4.0. Django 3.2, the next major release >>> will be an LTS. If we hold-off introducing the change until 4.0, we can >>> flag it as a breaking change in the 4.0 release notes, with big warnings, >>> allowing folks extra time to hang out on the previous LTS if they need it. >>> >>> What I wouldn't want to do is to bring the breaking change in in Django >>> 3.2, because we'll have a whole load of folks updating from the 2.2 LTS at >>> about the time when it goes End of Life, and with no warning, that'd be a >>> hard breaking change to throw on top of their other issues. >>> >>> We'd keep the shim in place for the entire 4.x series, removing in >>> Django 5.0 as per the deprecation policy >>> <https://docs.djangoproject.com/en/3.1/internals/release-process/#deprecation-policy> >>> . >>> >>> I think the advantages of doing it this way are two-fold: >>> >>> * We allow people to focus on the semantic breaking change (in folds) >>> separately from the code changes per se — the logic may have changed >>> slightly in these cases, but it'll still run. >>> * It looks easier to migrate Django's code vs branching on a new >>> setting. (I didn't think through exactly what that might look like, so >>> happy to see a PoC from anyone.) >>> >>> I'm more attached to the timeline (i.e. making the change after the next >>> LTS) than whether we use the deprecation shim or not, but can I ask others >>> to give this their thought too? >>> >>> Thanks again! >>> >>> Kind Regards, >>> >>> Carlton >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-develop...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/ce04a6b7-4409-4b20-ba30-4cd64dc0cabfn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/django-developers/ce04a6b7-4409-4b20-ba30-4cd64dc0cabfn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-develop...@googlegroups.com. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/e13e8ae2-5d43-e550-48a4-cb7ad6e699f6%40ganssle.io >>> >>> <https://groups.google.com/d/msgid/django-developers/e13e8ae2-5d43-e550-48a4-cb7ad6e699f6%40ganssle.io?utm_medium=email&utm_source=footer> >>> . >>> >>> -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-develop...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/b18754a4-c308-492d-b547-6b3c7cdc1442n%40googlegroups.com > > <https://groups.google.com/d/msgid/django-developers/b18754a4-c308-492d-b547-6b3c7cdc1442n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/043100f2-fd50-458a-9b31-c52128a534cbn%40googlegroups.com.
