New submission from Paul Ganssle <p.gans...@gmail.com>:
When examining the performance characteristics of pytz, I realized that pytz's
eager calculation of tzname, offset and DST gives it an implicit cache that
makes it much faster for repeated queries to .utcoffset(), .dst() and/or
.tzname() though the eager calculation means that it's slower to create an
aware datetime that never calculates those functions - see my blog post "pytz:
The Fastest Footgun in the West" [1].
I do not think that datetime should move to eager calculations (for one thing
it would be a pretty major change), but I did come up with a modest change that
can make it possible to implement a pythonic time zone provider without taking
the performance hit: introducing a small, optional, set-once cache for "time
zone index" onto the datetime object. The idea takes advantage of the fact that
essentially all time zones can be described by a very small number of (offset,
tzname, dst) combinations plus a function to describe which one applies. Time
zone implementations can store these offsets in one or more indexable
containers and implement a `tzidx(self, dt)` method returning the relevant
index as a function of the datetime. We would provide a per-datetime cache by
implementing a datetime.tzidx(self) method, which would be a memoized call to
`self.tzinfo.tzidx()`, like this (ignoring error handling - a more detailed
implementation can be found in the PoC PR):
def tzidx(self):
if self._tzidx != 0xff:
return self._tzidx
tzidx = self.tzinfo.tzidx(self)
if isinstance(tzidx, int) and 0 <= tzidx < 255:
self._tzidx = tzidx
return tzidx
And then `utcoffset(self, dt)`, `dst(self, dt)` and `tzname(self, dt)` could be
implemented in terms of `dt.tzidx()`! This interface would be completely
opt-in, and `tzinfo.tzidx` would have no default implementation.
Note that I have used 0xff as the signal value here - this is because I propose
that the `tzidx` cache be limited to *only* integers in the interval [0, 255),
with 255 reserved as the "not set" value. It is exceedingly unlikely that a
given time zone will have more than 255 distinct values in its index, and even
if it does, this implementation gracefully falls back to "every call is a cache
miss".
In my tests, using a single unsigned char for `tzidx` does not increase the
size of the `PyDateTime` struct, because it's using a byte that is currently
part of the alignment padding anyway. This same trick was used to minimize the
impact of `fold`, and I figure it's better to be conservative and enforce 0 <=
tzidx < 255, since we can only do it so many times.
The last thing I'd like to note is the problem of mutability - datetime objects
are supposed to be immutable, and this cache value actually mutates the
datetime struct! While it's true that the in-memory value of the datetime
changes, the fundamental concept of immutability is retained, since this does
not affect any of the qualities of the datetime observable via the public API.
In fact (and I hope this is not too much of a digression), it is already
unfortunately true that datetimes are more mutable than they would seem,
because nothing prevents `tzinfo` objects from returning different values on
subsequent calls to the timezone-lookup functions. What's worse, datetime's
hash implementation takes its UTC offset into account! In practice it's rare
for a tzinfo to ever return a different value for utcoffset(dt), but one
prominent example where this could be a problem is with something like
`dateutil.tz.tzlocal`, which is a local timezone object written in terms of the
`time` module's time zone information - which can change if the local timezone
information changes over the course of a program's run.
This change does not necessarily fix that problem or start enforcing
immutability of `utcoffset`, but it does encourage a design that is *less
susceptible* to these problems, since even if the return value of
`tzinfo.tzidx()` changes over time for some reason, that function would only be
called once per datetime.
I have an initial PoC for this implemented, and I've tested it out with an
accompanying implementation of `dateutil.tz` that makes use of it, it does
indeed make things much faster. I look forward to your comments.
1. https://blog.ganssle.io/articles/2018/03/pytz-fastest-footgun.html
----------
components: Library (Lib)
messages: 333503
nosy: p-ganssle
priority: normal
severity: normal
status: open
title: Add "time zone index" cache to datetime objects
type: enhancement
versions: Python 3.8
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue35723>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
