Re: [Python-Dev] Status on PEP-431 Timezones
On Thu, Apr 16, 2015 at 1:00 AM, Lennart Regebro rege...@gmail.com wrote: So because of this, perhaps we actually *should* change the internal representation to UTC, because that makes the issues I'm fighting with now so much simpler. (I'm currently trying to get arithmetic to do the right thing in all cases, which is crazy complicated). If I understand you correctly, then, an aware datetime would represent a unique instant in time (modulo relativity), coupled with some metadata stating what civil timezone it should be understood in terms of. This is the same as a PostgreSQL timestamp with time zone field, and IMO is a pretty reliable way to do things. So count me as +1 for this proposal. Bikeshed: Would arithmetic be based on UTC time or Unix time? It'd be more logical to describe it as adding six hours means adding six hours to the UTC time, but it'd look extremely odd when there's a leap second. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 11:10 AM, Chris Angelico ros...@gmail.com wrote: Bikeshed: Would arithmetic be based on UTC time or Unix time? It'd be more logical to describe it as adding six hours means adding six hours to the UTC time, but it'd look extremely odd when there's a leap second. It would ignore leap seconds. If you want to call that unix time or not is a matter of opinion. Hm. I guess the internal representation *could* be EPOCH + offset, and local times could be calculated properties, which could be cached (or possibly calculated at creation). //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
OK, so I just had a realization. Because we want some internal flag to tell if the datetime is in DST or not, the datetime pickle format will change. And the datetime pickle format changing is the biggest reason I had against changing the internal representation to UTC. So because of this, perhaps we actually *should* change the internal representation to UTC, because that makes the issues I'm fighting with now so much simpler. (I'm currently trying to get arithmetic to do the right thing in all cases, which is crazy complicated). We can add support to unpickle previous datetimes, but we won't be able to add forwards compatibility, meaning that pickles saved in Python 3.5 will not be unpicklable in Python 3.4. Please discuss. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 11:43 AM, Lennart Regebro rege...@gmail.com wrote: On Wed, Apr 15, 2015 at 11:10 AM, Chris Angelico ros...@gmail.com wrote: Bikeshed: Would arithmetic be based on UTC time or Unix time? It'd be more logical to describe it as adding six hours means adding six hours to the UTC time, but it'd look extremely odd when there's a leap second. It would ignore leap seconds. If you want to call that unix time or not is a matter of opinion. Hm. I guess the internal representation *could* be EPOCH + offset, and local times could be calculated properties, which could be cached (or possibly calculated at creation). In any case there wold probably need to be a PEP on that, and that means PEP 431 wouldn't make it into 3.5, unless somebody smarter than me want to take a shot at it. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 12:40 PM, Isaac Schwabacher ischwabac...@wisc.edu wrote: I am on the fence about EPOCH + offset as an internal representation. On the one hand, it is conceptually cleaner than the horrible byte-packing that exists today, but on the other hand, it has implications for future implementations of leap-second-awareness. For instance, offset measures the difference between the local time and UTC. But is it honest-to-goodness leap-second aware UTC, or is it really Unix time? This could be solved by giving each tzinfo a pointer to the UTC from which it is offset, but that sounds like a can of worms we don't want to open. But if don't and we store EPOCH + offset, we can't add leap second awareness to UTC without corrupting all persisted aware datetimes. That's true, with separate values like there is now we can easily allow 23:59:60 as a timestamp during leap seconds. I'm not entirely sure it makes a big difference though, I don't think we ever wants to deal with leap seconds by default. I don't think we ever want the standard arithmetic deal with leap seconds anyway. datetime(2012, 6, 30, 23, 30) + timedelta(seconds=3600) returning datetime(2012, 7, 1, 0, 29, 59) I guess leap second implementations should rather have special functions for arithmethics that deal with this. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On 15-04-15, Lennart Regebro wrote: On Wed, Apr 15, 2015 at 11:10 AM, Chris Angelico ros...@gmail.com wrote: Bikeshed: Would arithmetic be based on UTC time or Unix time? It'd be more logical to describe it as adding six hours means adding six hours to the UTC time, but it'd look extremely odd when there's a leap second. It would ignore leap seconds. If you want to call that unix time or not is a matter of opinion. Hm. I guess the internal representation *could* be EPOCH + offset, and local times could be calculated properties, which could be cached (or possibly calculated at creation). I am on the fence about EPOCH + offset as an internal representation. On the one hand, it is conceptually cleaner than the horrible byte-packing that exists today, but on the other hand, it has implications for future implementations of leap-second-awareness. For instance, offset measures the difference between the local time and UTC. But is it honest-to-goodness leap-second aware UTC, or is it really Unix time? This could be solved by giving each tzinfo a pointer to the UTC from which it is offset, but that sounds like a can of worms we don't want to open. But if don't and we store EPOCH + offset, we can't add leap second awareness to UTC without corrupting all persisted aware datetimes. Also, I didn't mention this before because I figured people were getting sick of my dumb idea, but another advantage of always caching the offset is that you can detect future datetimes that have been corrupted by zoneinfo changes. You need both absolute time and offset to be able to do this. ijs ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On 15-04-15, Lennart Regebro wrote: On Wed, Apr 15, 2015 at 12:40 PM, Isaac Schwabacher ischwabac...@wisc.edu wrote: I am on the fence about EPOCH + offset as an internal representation. On the one hand, it is conceptually cleaner than the horrible byte-packing that exists today, but on the other hand, it has implications for future implementations of leap-second-awareness. For instance, offset measures the difference between the local time and UTC. But is it honest-to-goodness leap-second aware UTC, or is it really Unix time? This could be solved by giving each tzinfo a pointer to the UTC from which it is offset, but that sounds like a can of worms we don't want to open. But if don't and we store EPOCH + offset, we can't add leap second awareness to UTC without corrupting all persisted aware datetimes. That's true, with separate values like there is now we can easily allow 23:59:60 as a timestamp during leap seconds. I'm not entirely sure it makes a big difference though, I don't think we ever wants to deal with leap seconds by default. I don't think we ever want the standard arithmetic deal with leap seconds anyway. datetime(2012, 6, 30, 23, 30) + timedelta(seconds=3600) returning datetime(2012, 7, 1, 0, 29, 59) I guess leap second implementations should rather have special functions for arithmethics that deal with this. You need relative timedeltas to mitigate the pain of leap seconds, yes. But as soon as you have timedeltas that are capable of representing this number of seconds into the next minute (one minute) as opposed to sixty seconds, this isn't so much of a problem. Though of course subtraction will (and should!) continue to yield timedelta(seconds=3601). From my perspective, the issue is more that the stdlib shouldn't rule out leap seconds. It's reasonable enough to expect users who actually want them to write appropriate tzinfo and timedelta classes, but we don't want to make that impossible the way the old tzinfo interface made DST-aware time zones impossible by insisting that implementers implement a function that wasn't mathematically a function. I need to think about this more before I can get a real rant going. ijs ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
Lennart Regebro rege...@gmail.com writes:
OK, so I realized another thing today, and that is that arithmetic
doesn't necessarily round trip.
For example, 2002-10-27 01:00 US/Eastern comes both in DST and STD.
But 2002-10-27 01:00 US/Eastern STD minus two days is 2002-10-25 01:00
US/Eastern DST
two days is ambiguous here. It is incorrect if you mean 48 hours (the
difference is 49 hours):
#!/usr/bin/env python3
from datetime import datetime, timedelta
import pytz
tz = pytz.timezone('US/Eastern')
then_isdst = False # STD
then = tz.localize(datetime(2002, 10, 27, 1), is_dst=then_isdst)
now = tz.localize(datetime(2002, 10, 25, 1), is_dst=None) # no utc transition
print((then - now) // timedelta(hours=1))
# - 49
However, 2002-10-25 01:00 US/Eastern DST plus two days is 2002-10-27
01:00 US/Eastern, but it is ambiguous if you want DST or not DST.
It is not ambiguous if you know what two days *in your particular
application* should mean (`day+2` vs. +48h exactly):
print(tz.localize(now.replace(tzinfo=None) + timedelta(2), is_dst=then_isdst))
# - 2002-10-27 01:00:00-05:00 # +49h
print(tz.normalize(now + timedelta(2))) # +48h
# - 2002-10-27 01:00:00-04:00
Here's a simple mental model that can be used for date arithmetics:
- naive datetime + timedelta(2) == same time, elapsed hours unknown
- aware utc datetime + timedelta(2) == same time, +48h
- aware datetime with timezone that may have different utc offsets at
different times + timedelta(2) == unknown time, +48h
unknown means that you can't tell without knowning the specific
timezone.
It ignores leap seconds.
The 3rd case behaves *as if* the calculations are performed using these
steps (the actual implementation may be different):
1. convert an aware datetime
object to utc (dt.astimezone(pytz.utc))
2. do the simple arithmetics using utc time
3. convert the result to the original pytz timezone (utc_dt.astimezone(tz))
you don't need `.localize()`, `.normalize()` calls here.
And you can't pass in a is_dst flag to __add__, so the arithmatic must
just pick one, and the sensible one is to keep to the same DST.
That means that:
tz = get_timezone('US/Eastern')
dt = datetimedatetime(2002, 10, 27, 1, 0, tz=tz, is_dst=False)
dt2 = dt - 420 + 420
assert dt == dt2
Will fail, which will be unexpected for most people.
I think there is no way around this, but I thought I should flag for
it. This is a good reason to do all your date time arithmetic in UTC.
//Lennart
It won't fail:
from datetime import datetime, timedelta
import pytz
tz = pytz.timezone('US/Eastern')
dt = tz.localize(datetime(2002, 10, 27, 1), is_dst=False)
delta = timedelta(seconds=420)
assert dt == tz.normalize(tz.normalize(dt - delta) + delta)
The only reason `tz.normalize()` is used so that tzinfo would be correct
for the resulting datetime object; it does not affect the comparison otherwise:
assert dt == (dt - delta + delta) #XXX tzinfo may be incorrect
assert dt == tz.normalize(dt - delta + delta) # correct tzinfo for the final
result
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
Isaac Schwabacher ischwabac...@wisc.edu writes:
...
I know that you can do datetime.now(tz), and you can do datetime(2013,
11, 3, 1, 30, tzinfo=zoneinfo('America/Chicago')), but not being able
to add a time zone to an existing naive datetime is painful (and
strptime doesn't even let you pass in a time zone).
`.now(tz)` is correct. `datetime(..., tzinfo=tz)`) is wrong: if tz is a
pytz timezone then you may get a wrong tzinfo (LMT), you should use
`tz.localize(naive_dt, is_dst=False|True|None)` instead.
...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On 15-04-15, Akira Li 4kir4...@gmail.com wrote:
Isaac Schwabacher ischwabac...@wisc.edu writes:
...
I know that you can do datetime.now(tz), and you can do datetime(2013,
11, 3, 1, 30, tzinfo=zoneinfo('America/Chicago')), but not being able
to add a time zone to an existing naive datetime is painful (and
strptime doesn't even let you pass in a time zone).
`.now(tz)` is correct. `datetime(..., tzinfo=tz)`) is wrong: if tz is a
pytz timezone then you may get a wrong tzinfo (LMT), you should use
`tz.localize(naive_dt, is_dst=False|True|None)` instead.
The whole point of this thread is to finalize PEP 431, which fixes the problem
for which `localize()` and `normalize()` are workarounds. When this is done,
`datetime(..., tzinfo=tz)` will be correct.
ijs
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Aware datetime from naive local time Was: Status on PEP-431 Timezones
Alexander Belopolsky alexander.belopol...@gmail.com writes:
Sorry for a truncated message. Please scroll past the quoted portion.
On Thu, Apr 9, 2015 at 10:21 PM, Alexander Belopolsky
alexander.belopol...@gmail.com wrote:
On Thu, Apr 9, 2015 at 4:51 PM, Isaac Schwabacher ischwabac...@wisc.edu
wrote:
Well, you are right, but at least we do have a localtime utility
hidden in the email package:
from datetime import *
from email.utils import localtime
print(localtime(datetime.now()))
2015-04-09 15:19:12.84-04:00
You can read http://bugs.python.org/issue9527 for the reasons it
did not make into datetime.
But that's restricted to the system time zone. Nothing good ever
comes from the system time zone...
Let's solve one problem at a time. ...
PEP 431 proposes to import zoneinfo into the stdlib, ...
I am changing the subject so that we can focus on one question without
diverting to PEP-size issues that are better suited for python ideas.
I would like to add a functionality to the datetime module that would
solve a seemingly simple problem: given a naive datetime instance assumed
to be in local time, construct the corresponding aware datetime object with
tzinfo set to an appropriate fixed offset datetime.timezone instance.
Python 3 has this functionality implemented in the email package since
version 3.3, and it appears to work well even
in the ambiguous hour
from email.utils import localtime
from datetime import datetime
localtime(datetime(2014,11,2,1,30)).strftime('%c %z %Z')
'Sun Nov 2 01:30:00 2014 -0400 EDT'
localtime(datetime(2014,11,2,1,30), isdst=0).strftime('%c %z %Z')
'Sun Nov 2 01:30:00 2014 -0500 EST'
However, in a location with a more interesting history, you can get a
situation that
would look like this in the zoneinfo database:
$ zdump -v -c 1992 Europe/Kiev
...
Europe/Kiev Sat Mar 24 22:59:59 1990 UTC = Sun Mar 25 01:59:59 1990 MSK
isdst=0
Europe/Kiev Sat Mar 24 23:00:00 1990 UTC = Sun Mar 25 03:00:00 1990 MSD
isdst=1
Europe/Kiev Sat Jun 30 21:59:59 1990 UTC = Sun Jul 1 01:59:59 1990 MSD
isdst=1
Europe/Kiev Sat Jun 30 22:00:00 1990 UTC = Sun Jul 1 01:00:00 1990 EEST
isdst=1
Europe/Kiev Sat Sep 28 23:59:59 1991 UTC = Sun Sep 29 02:59:59 1991 EEST
isdst=1
Europe/Kiev Sun Sep 29 00:00:00 1991 UTC = Sun Sep 29 02:00:00 1991 EET
isdst=0
...
Look what happened on July 1, 1990. At 2 AM, the clocks in Ukraine were
moved back one hour. So times like 01:30 AM happened twice there on that
day. Let's see how Python handles this situation
$ TZ=Europe/Kiev python3
from email.utils import localtime
from datetime import datetime
localtime(datetime(1990,7,1,1,30)).strftime('%c %z %Z')
'Sun Jul 1 01:30:00 1990 +0400 MSD'
So far so good, I've got the first of the two 01:30AM's. But what if I
want the other 01:30AM? Well,
localtime(datetime(1990,7,1,1,30), isdst=0).strftime('%c %z %Z')
'Sun Jul 1 01:30:00 1990 +0300 EEST'
gives me the other 01:30AM, but it is counter-intuitive: I have to ask
for the standard (winter) time to get the daylight savings (summer) time.
It looks incorrect. Here's the corresponding pytz code:
from datetime import datetime
import pytz
tz = pytz.timezone('Europe/Kiev')
print(tz.localize(datetime(1990, 7, 1, 1, 30), is_dst=False).strftime('%c %z
%Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
print(tz.localize(datetime(1990, 7, 1, 1, 30), is_dst=True).strftime('%c %z
%Z'))
# - Sun Jul 1 01:30:00 1990 +0400 MSD
See also Enhance support for end-of-DST-like ambiguous time [1]
[1] https://bugs.launchpad.net/pytz/+bug/1378150
`email.utils.localtime()` is broken:
from datetime import datetime
from email.utils import localtime
print(localtime(datetime(1990, 7, 1, 1, 30)).strftime('%c %z %Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
print(localtime(datetime(1990, 7, 1, 1, 30), isdst=0).strftime('%c %z %Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
print(localtime(datetime(1990, 7, 1, 1, 30), isdst=1).strftime('%c %z %Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
print(localtime(datetime(1990, 7, 1, 1, 30), isdst=-1).strftime('%c %z %Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
Versions:
$ ./python -V
Python 3.5.0a3+
$ dpkg -s tzdata | grep -i version
Version: 2015b-0ubuntu0.14.04
The uncertainty about how to deal with the repeated hour was the reason why
email.utils.localtime-like interface did not make it to the datetime
module.
repeated hour (time jumps back) can be treated like a end-of-DST
transition, to resolve ambiguities [1].
The main objection to the isdst flag was that in most situations,
determining whether DST is in effect is as hard as finding the UTC offset,
so reducing the problem of finding the UTC offset to the one of finding the
value for isdst does not solve much.
I now realize that the problem is simply in the name for the flag. While
we cannot often tell what isdst
Re: [Python-Dev] Status on PEP-431 Timezones
On 15 April 2015 at 17:00, Lennart Regebro rege...@gmail.com wrote: OK, so I just had a realization. Because we want some internal flag to tell if the datetime is in DST or not, the datetime pickle format will change. And the datetime pickle format changing is the biggest reason I had against changing the internal representation to UTC. So because of this, perhaps we actually *should* change the internal representation to UTC, because that makes the issues I'm fighting with now so much simpler. (I'm currently trying to get arithmetic to do the right thing in all cases, which is crazy complicated). Huh. I didn't think you would need to change any arithmetic (but haven't looked at this for quite some time). You can already add or subtract timedeltas to timezone aware datetime instances. The problem with the existing implementation is the tzinfo instance does not have enough information to do correct conversions when the time is ambiguous, so it has to guess. With the addition of the is_dst hint to the datetime instance, it will no longer need to guess. Arithmetic remains 'add the timedelta to the naive datetime, and then punt it to the tzinfo to make any necessary adjustments' and I thought this would not need to be changed at all. We can add support to unpickle previous datetimes, but we won't be able to add forwards compatibility, meaning that pickles saved in Python 3.5 will not be unpicklable in Python 3.4. I don't think this can be avoided entirely. Any ideas I can come up with that might help are worse than requiring devs to convert their datetimes to strings in the rare case they need their 3.5 pickles read with 3.4. -- Stuart Bishop stu...@stuartbishop.net http://www.stuartbishop.net/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Thu, Apr 16, 2015 at 1:43 AM, Lennart Regebro rege...@gmail.com wrote: On Wed, Apr 15, 2015 at 11:10 AM, Chris Angelico ros...@gmail.com wrote: Bikeshed: Would arithmetic be based on UTC time or Unix time? It'd be more logical to describe it as adding six hours means adding six hours to the UTC time, but it'd look extremely odd when there's a leap second. It would ignore leap seconds. If you want to call that unix time or not is a matter of opinion. Hm. I guess the internal representation *could* be EPOCH + offset, and local times could be calculated properties, which could be cached (or possibly calculated at creation). I was just talking about leap seconds, here (which Unix time ignores), not about the internal representation, which is an implementation detail. If a timedelta is represented as a number of seconds, then adding six hours really means adding 6*3600 seconds, and most people would be VERY surprised if one of those is consumed by a leap second; but it ought at least to be acknowledged in the docs. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
Alexander Belopolsky alexander.belopol...@gmail.com writes: On Wed, Apr 8, 2015 at 3:57 PM, Isaac Schwabacher ischwabac...@wisc.edu wrote: On 15-04-08, Alexander Belopolsky wrote: With datetime, we also have a problem that POSIX APIs don't have to deal with: local time arithmetics. What is t + timedelta(1) when t falls on the day before DST change? How would you set the isdst flag in the result? It's whatever time comes 60*60*24 seconds after t in the same time zone, because the timedelta class isn't expressive enough to represent anything but absolute time differences (nor should it be, IMO). This is not what most uses expect. The expect datetime(y, m, d, 12, tzinfo=New_York) + timedelta(1) to be datetime(y, m, d+1, 12, tzinfo=New_York) It is incorrect. If you want d+1 for +timedelta(1); use a **naive** datetime. Otherwise +timedelta(1) is +24h: tomorrow = tz.localize(aware_dt.replace(tzinfo=None) + timedelta(1), is_dst=None) dt_plus24h = tz.normalize(aware_dt + timedelta(1)) # +24h *tomorrow* and *aware_dt* have the *same* time but it is unknown how many hours have passed if the utc offset has changed in between. *dt_plus24h* may have a different time but there are exactly 24 hours have passed between *dt_plush24* and *aware_dt* http://stackoverflow.com/questions/441147/how-can-i-subtract-a-day-from-a-python-date ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 3:23 PM, Stuart Bishop stu...@stuartbishop.net wrote: Huh. I didn't think you would need to change any arithmetic Not really, the problem is in keeping the date normalized after each call, and doing so the right way. Arithmetic remains 'add the timedelta to the naive datetime, and then punt it to the tzinfo to make any necessary adjustments' and I thought this would not need to be changed at all. Just punting it to tzinfo to make adjustments, ie effectively just doing what normalize() does creates infinite recursion as there is more arithmetic in there, so it's not quite that simple. I don't think this can be avoided entirely. Any ideas I can come up with that might help are worse than requiring devs to convert their datetimes to strings in the rare case they need their 3.5 pickles read with 3.4. Pickle forward compatibility isn't really expected anyway... ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 Stable ABI
I don't see any obvious issues, but there may be some that don't need to be marked stable. Given that a mismatch here will cause build errors for users, I'm +1 on checking this in. Cheers, Steve Top-posted from my Windows Phone From: Zachary Waremailto:zachary.ware+py...@gmail.com Sent: 4/13/2015 17:29 To: Python-Devmailto:python-dev@python.org Subject: [Python-Dev] Python3 Stable ABI In issue23903, I've created a script that will produce PC/python3.def by scraping the header files in Include. There are are many many discrepencies between what my script generates and what is currently in the repository (diff below), but in every case I've checked the script has been right: what the script finds is actually exported as part of the limited API, but due to not being in the .def file it's not actually exported from python3.dll. Almost all of the differences are things that the script found that weren't present, but there are a couple things going the other way. The point of this message is to ask everybody who maintains anything in C to take a look through and make sure everything in their area is properly guarded (or not) by Py_LIMITED_API. Alternately, if somebody can find a bug in my script and brain that's finding too much stuff, that would be great too. Ideally, after this is all settled I'd like to add the script to both the Makefile and the Windows build system, such that PC/python3.def is always kept up to date and flags changes that weren't meant to be made. Regards, -- Zach (I'm afraid Gmail might mangle this beyond recognition, you can find the diff at http://bugs.python.org/review/23903/diff/14549/PC/python3.def if it does.) diff -r 24f2c0279120 PC/python3.def --- a/PC/python3.defMon Apr 13 15:51:59 2015 -0500 +++ b/PC/python3.defMon Apr 13 16:10:34 2015 -0500 @@ -1,13 +1,15 @@ ; This file specifies the import forwarding for python3.dll ; It is used when building python3dll.vcxproj +; Generated by python3defgen.py, DO NOT modify directly! LIBRARYpython3 EXPORTS + PyAST_FromNode=python35.PyAST_FromNode + PyAST_FromNodeObject=python35.PyAST_FromNodeObject + PyAST_Validate=python35.PyAST_Validate PyArg_Parse=python35.PyArg_Parse PyArg_ParseTuple=python35.PyArg_ParseTuple PyArg_ParseTupleAndKeywords=python35.PyArg_ParseTupleAndKeywords PyArg_UnpackTuple=python35.PyArg_UnpackTuple - PyArg_VaParse=python35.PyArg_VaParse - PyArg_VaParseTupleAndKeywords=python35.PyArg_VaParseTupleAndKeywords PyArg_ValidateKeywordArguments=python35.PyArg_ValidateKeywordArguments PyBaseObject_Type=python35.PyBaseObject_Type DATA PyBool_FromLong=python35.PyBool_FromLong @@ -39,7 +41,6 @@ PyCFunction_GetFlags=python35.PyCFunction_GetFlags PyCFunction_GetFunction=python35.PyCFunction_GetFunction PyCFunction_GetSelf=python35.PyCFunction_GetSelf - PyCFunction_New=python35.PyCFunction_New PyCFunction_NewEx=python35.PyCFunction_NewEx PyCFunction_Type=python35.PyCFunction_Type DATA PyCallIter_New=python35.PyCallIter_New @@ -58,6 +59,7 @@ PyCapsule_SetPointer=python35.PyCapsule_SetPointer PyCapsule_Type=python35.PyCapsule_Type DATA PyClassMethodDescr_Type=python35.PyClassMethodDescr_Type DATA + PyCmpWrapper_Type=python35.PyCmpWrapper_Type DATA PyCodec_BackslashReplaceErrors=python35.PyCodec_BackslashReplaceErrors PyCodec_Decode=python35.PyCodec_Decode PyCodec_Decoder=python35.PyCodec_Decoder @@ -68,6 +70,7 @@ PyCodec_IncrementalEncoder=python35.PyCodec_IncrementalEncoder PyCodec_KnownEncoding=python35.PyCodec_KnownEncoding PyCodec_LookupError=python35.PyCodec_LookupError + PyCodec_NameReplaceErrors=python35.PyCodec_NameReplaceErrors PyCodec_Register=python35.PyCodec_Register PyCodec_RegisterError=python35.PyCodec_RegisterError PyCodec_ReplaceErrors=python35.PyCodec_ReplaceErrors @@ -122,6 +125,7 @@ PyErr_Fetch=python35.PyErr_Fetch PyErr_Format=python35.PyErr_Format PyErr_FormatV=python35.PyErr_FormatV + PyErr_GetExcInfo=python35.PyErr_GetExcInfo PyErr_GivenExceptionMatches=python35.PyErr_GivenExceptionMatches PyErr_NewException=python35.PyErr_NewException PyErr_NewExceptionWithDoc=python35.PyErr_NewExceptionWithDoc @@ -132,14 +136,25 @@ PyErr_PrintEx=python35.PyErr_PrintEx PyErr_ProgramText=python35.PyErr_ProgramText PyErr_Restore=python35.PyErr_Restore + PyErr_SetExcFromWindowsErr=python35.PyErr_SetExcFromWindowsErr + PyErr_SetExcFromWindowsErrWithFilename=python35.PyErr_SetExcFromWindowsErrWithFilename + PyErr_SetExcFromWindowsErrWithFilenameObject=python35.PyErr_SetExcFromWindowsErrWithFilenameObject + PyErr_SetExcFromWindowsErrWithFilenameObjects=python35.PyErr_SetExcFromWindowsErrWithFilenameObjects + PyErr_SetExcInfo=python35.PyErr_SetExcInfo + PyErr_SetExcWithArgsKwargs=python35.PyErr_SetExcWithArgsKwargs PyErr_SetFromErrno=python35.PyErr_SetFromErrno PyErr_SetFromErrnoWithFilename=python35.PyErr_SetFromErrnoWithFilename
Re: [Python-Dev] Status on PEP-431 Timezones
Isaac Schwabacher ischwabac...@wisc.edu writes:
On 15-04-15, Akira Li 4kir4...@gmail.com wrote:
Isaac Schwabacher ischwabac...@wisc.edu writes:
...
I know that you can do datetime.now(tz), and you can do datetime(2013,
11, 3, 1, 30, tzinfo=zoneinfo('America/Chicago')), but not being able
to add a time zone to an existing naive datetime is painful (and
strptime doesn't even let you pass in a time zone).
`.now(tz)` is correct. `datetime(..., tzinfo=tz)`) is wrong: if tz is a
pytz timezone then you may get a wrong tzinfo (LMT), you should use
`tz.localize(naive_dt, is_dst=False|True|None)` instead.
The whole point of this thread is to finalize PEP 431, which fixes the
problem for which `localize()` and `normalize()` are workarounds. When
this is done, `datetime(..., tzinfo=tz)` will be correct.
ijs
The input time is ambiguous. Even if we assume PEP 431 is implemented in
some form, your code is still missing isdst parameter (or the
analog). PEP 431 won't fix it; it can't resolve the ambiguity by
itself. Notice is_dst paramter in the `tz.localize()` call (current
API).
.now(tz) works even during end-of-DST transitions (current API) when the
local time is ambiguous.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Aware datetime from naive local time Was: Status on PEP-431 Timezones
Alexander Belopolsky alexander.belopol...@gmail.com writes:
...
For most world locations past discontinuities are fairly well documented
for at least a century and future changes are published with at least 6
months lead time.
It is important to note that the different versions of the tz database
may lead to different tzinfo (utc offset, tzname) even for *past* dates.
i.e., (lt, tzid, isdst) is not enough because the result for (lt,
tzid(2015b), isdst) may be different from (lt, tzid(X), isdst)
where
lt = local time e.g., naive datetime
tzid = timezone from the tz database e.g., Europe/Kiev
isdst = a boolean flag for disambiguation
X != 2015b
In other words, a fixed utc offset might not be sufficient even for past
dates.
...
Moreover, a program that rejects invalid times on input, but stores them
for a long time may see its database silently corrupted after a zoneinfo
update.
Now it is time to make specific proposal. I would like to extend
datetime.astimezone() method to work on naive datetime instances. Such
instances will be assumed to be in local time and discontinuities will be
handled as follows:
1. wall(t) == lt has a single solution. This is the trivial case and
lt.astimezone(utc) and lt.astimezone(utc, which=i) for i=0,1 should return
that solution.
2. wall(t) == lt has two solutions t1 and t2 such that t1 t2. In this
case lt.astimezone(utc) == lt.astimezone(utc, which=0) == t1 and
lt.astimezone(utc, which=1) == t2.
In pytz terms: `which = not isdst` (end-of-DST-like transition: isdst
changes from True to False in the direction of utc time).
It resolves AmbiguousTimeError raised by `tz.localize(naive, is_dst=None)`.
3. wall(t) == lt has no solution. This happens when there is UTC time t0
such that wall(t0) lt and wall(t0+epsilon) lt (a positive discontinuity
at time t0). In this case lt.astimezone(utc) should return t0 + lt -
wall(t0). I.e., we ignore the discontinuity and extend wall(t) linearly
past t0. Obviously, in this case the invariant wall(lt.astimezone(utc)) ==
lt won't hold. The which flag should be handled as follows:
lt.astimezone(utc) == lt.astimezone(utc, which=0) and lt.astimezone(utc,
which=0) == t0 + lt - wall(t0+eps).
It is inconsistent with the previous case: here `which = isdst` but
`which = not isdst` above.
`lt.astimezone(utc, which=0) == t0 + lt - wall(t0+eps)` corresponds to:
result = tz.normalize(tz.localize(lt, isdst=False))
i.e., `which = isdst` (t0 is at the start of DST and therefore isdst
changes from False to True).
It resolves NonExistentTimeError raised by `tz.localize(naive,
is_dst=None)`. start-of-DST-like transition (Spring forward).
For example,
from datetime import datetime, timedelta
import pytz
tz = pytz.timezone('America/New_York')
# 2am -- non-existent time
print(tz.normalize(tz.localize(datetime(2015, 3, 8, 2), is_dst=False)))
# - 2015-03-08 03:00:00-04:00 # after the jump (wall(t0+eps))
print(tz.localize(datetime(2015, 3, 8, 3), is_dst=None))
# - 2015-03-08 03:00:00-04:00 # same time, unambiguous
# 2:01am -- non-existent time
print(tz.normalize(tz.localize(datetime(2015, 3, 8, 2, 1), is_dst=False)))
# - 2015-03-08 03:01:00-04:00
print(tz.localize(datetime(2015, 3, 8, 3, 1), is_dst=None))
# - 2015-03-08 03:01:00-04:00 # same time, unambiguous
# 2:59am non-existent time
dt = tz.normalize(tz.localize(datetime(2015, 3, 8, 2, 59), is_dst=True))
print(dt)
# - 2015-03-08 01:59:00-05:00 # before the jump (wall(t0-eps))
print(tz.normalize(dt + timedelta(minutes=1)))
# - 2015-03-08 03:00:00-04:00
With the proposed features in place, one can use the naive code
t = lt.astimezone(utc)
and get predictable behavior in all cases and no crashes.
A more sophisticated program can be written like this:
t1 = lt.astimezone(utc, which=0)
t2 = lt.astimezone(utc, which=1)
if t1 == t2:
t = t1
elif t2 t1:
# ask the user to pick between t1 and t2 or raise
AmbiguousLocalTimeError
else:
t = t1
# warn the user that time was invalid and changed or raise
InvalidLocalTimeError
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On 15-04-15, Akira Li 4kir4...@gmail.com wrote:
Isaac Schwabacher ischwabac...@wisc.edu writes:
On 15-04-15, Akira Li 4kir4...@gmail.com wrote:
Isaac Schwabacher ischwabac...@wisc.edu writes:
...
I know that you can do datetime.now(tz), and you can do datetime(2013,
11, 3, 1, 30, tzinfo=zoneinfo('America/Chicago')), but not being able
to add a time zone to an existing naive datetime is painful (and
strptime doesn't even let you pass in a time zone).
`.now(tz)` is correct. `datetime(..., tzinfo=tz)`) is wrong: if tz is a
pytz timezone then you may get a wrong tzinfo (LMT), you should use
`tz.localize(naive_dt, is_dst=False|True|None)` instead.
The whole point of this thread is to finalize PEP 431, which fixes the
problem for which `localize()` and `normalize()` are workarounds. When
this is done, `datetime(..., tzinfo=tz)` will be correct.
ijs
The input time is ambiguous. Even if we assume PEP 431 is implemented in
some form, your code is still missing isdst parameter (or the
analog). PEP 431 won't fix it; it can't resolve the ambiguity by
itself. Notice is_dst paramter in the `tz.localize()` call (current
API).
...yeah, I forgot to throw that in there. It was supposed to be there all
along. Nothing to see here, move along.
.now(tz) works even during end-of-DST transitions (current API) when the
local time is ambiguous.
I know that. That's what I was complaining about-- I was trying to talk about
how astimezone() was going to be inadequate even after the PEP was implemented
because it couldn't turn naive datetimes into aware ones, and people were
giving examples that started with aware datetimes generated by now(tz), which
completely went around the point I was trying to make. But it looks like
astimezone() is going to grow an is_dst parameter, and everything will be OK.
ijs
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Aware datetime from naive local time Was: Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 4:46 PM, Akira Li 4kir4...@gmail.com wrote:
Look what happened on July 1, 1990. At 2 AM, the clocks in Ukraine were
moved back one hour. So times like 01:30 AM happened twice there on that
day. Let's see how Python handles this situation
$ TZ=Europe/Kiev python3
from email.utils import localtime
from datetime import datetime
localtime(datetime(1990,7,1,1,30)).strftime('%c %z %Z')
'Sun Jul 1 01:30:00 1990 +0400 MSD'
So far so good, I've got the first of the two 01:30AM's. But what if I
want the other 01:30AM? Well,
localtime(datetime(1990,7,1,1,30), isdst=0).strftime('%c %z %Z')
'Sun Jul 1 01:30:00 1990 +0300 EEST'
gives me the other 01:30AM, but it is counter-intuitive: I have to ask
for the standard (winter) time to get the daylight savings (summer)
time.
It looks incorrect. Here's the corresponding pytz code:
from datetime import datetime
import pytz
tz = pytz.timezone('Europe/Kiev')
print(tz.localize(datetime(1990, 7, 1, 1, 30),
is_dst=False).strftime('%c %z %Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
print(tz.localize(datetime(1990, 7, 1, 1, 30), is_dst=True).strftime('%c
%z %Z'))
# - Sun Jul 1 01:30:00 1990 +0400 MSD
See also Enhance support for end-of-DST-like ambiguous time [1]
[1] https://bugs.launchpad.net/pytz/+bug/1378150
`email.utils.localtime()` is broken:
If you think there is a bug in email.utils.localtime - please open an issue
at bugs.python.org.
from datetime import datetime
from email.utils import localtime
print(localtime(datetime(1990, 7, 1, 1, 30)).strftime('%c %z %Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
print(localtime(datetime(1990, 7, 1, 1, 30), isdst=0).strftime('%c %z
%Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
print(localtime(datetime(1990, 7, 1, 1, 30), isdst=1).strftime('%c %z
%Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
print(localtime(datetime(1990, 7, 1, 1, 30), isdst=-1).strftime('%c %z
%Z'))
# - Sun Jul 1 01:30:00 1990 +0300 EEST
Versions:
$ ./python -V
Python 3.5.0a3+
$ dpkg -s tzdata | grep -i version
Version: 2015b-0ubuntu0.14.04
The uncertainty about how to deal with the repeated hour was the reason
why
email.utils.localtime-like interface did not make it to the datetime
module.
repeated hour (time jumps back) can be treated like a end-of-DST
transition, to resolve ambiguities [1].
I don't understand what you are complaining about. It is quite possible
that pytz uses is_dst flag differently from the way email.utils.localtime
uses isdst.
I was not able to find a good description of what is_dst means in pytz, but
localtime's isdst is documented as follows:
a positive or zero value for *isdst* causes localtime to
presume initially that summer time (for example, Daylight Saving Time)
is or is not (respectively) in effect for the specified time.
Can you demonstrate that email.utils.localtime does not behave as
documented?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 5:28 PM, Stuart Bishop stu...@stuartbishop.net wrote: On 15 April 2015 at 21:51, Lennart Regebro rege...@gmail.com wrote: On Wed, Apr 15, 2015 at 3:23 PM, Stuart Bishop stu...@stuartbishop.net wrote: Just punting it to tzinfo to make adjustments, ie effectively just doing what normalize() does creates infinite recursion as there is more arithmetic in there, so it's not quite that simple. This sounds familiar. Its infinite recursion if the tzinfo does its calculations using localized datetimes. If the tzinfo is stripped for the calculations, there is no tzinfo to recurse into. At least this was how I hoped it would work, and it sucks if it doesn't. You could be right that using the UTC representation internally for datetimes with a tzinfo makes the most sense. There is no infinite recursion in the way datetime module deals with zone conversions. However, implementors of tzinfo subclasses often overlook the fact that datetime module design mandates specific rules for what utcoffset() should return for the missing and ambiguous hours. Granted, the relevant section in the manual [1] is not an easy read and in fact for a long time that documentation itself was displaying a buggy implementation of the LocalTimezone class. [2] Understanding how the design works requires a bit of algebra [3], but I strongly recommend that anyone trying to improve the timezones support in the datetime module, print out those 200 lines of comments and go through them with a pencil following the proofs. Note that one of the key assumptions [3.2] in that write-up does not hold in real life. The assumption is that standard time offset does not depend on the point in time. However, I do believe that this assumption can be relaxed without invalidating the main result. I believe we can still have unambiguous fromutc() as long as standard time offset does not change too often. Basically, if we (generously) allow utcoffset to vary from -24h to +24h, then a sane zone can be defined as the one where utcoffset changes at most once in any 48 hour period. If I am right about this and the algebra works out, then we don't need to change datetime module design to properly support all world timezones. [1] https://docs.python.org/3/library/datetime.html#datetime.tzinfo.fromutc [2] http://bugs.python.org/issue9063 [3] https://hg.python.org/cpython/file/132b5376bf34/Lib/datetime.py#l1935 [3.2] https://hg.python.org/cpython/file/132b5376bf34/Lib/datetime.py#l1948 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On 15 April 2015 at 21:51, Lennart Regebro rege...@gmail.com wrote: On Wed, Apr 15, 2015 at 3:23 PM, Stuart Bishop stu...@stuartbishop.net wrote: Just punting it to tzinfo to make adjustments, ie effectively just doing what normalize() does creates infinite recursion as there is more arithmetic in there, so it's not quite that simple. This sounds familiar. Its infinite recursion if the tzinfo does its calculations using localized datetimes. If the tzinfo is stripped for the calculations, there is no tzinfo to recurse into. At least this was how I hoped it would work, and it sucks if it doesn't. You could be right that using the UTC representation internally for datetimes with a tzinfo makes the most sense. -- Stuart Bishop stu...@stuartbishop.net http://www.stuartbishop.net/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
Yeah, I just realized this. As long as you use timedelta, the
difference is of course not one day, but 24 hours. That solves the
problem, but it is surprising in other ways.
In US/Eastern datetime.datetime(2002, 10, 27, 1, 0) -
datetime.timedelta(1) needs to become datetime.datetime(2002, 10, 26,
2, 0)
(Note the hour change)
I was thinking in calendrial arithmetic, which the datetime module
doesn't need to care about.
On Wed, Apr 15, 2015 at 12:59 AM, Stuart Bishop stu...@stuartbishop.net wrote:
On 14 April 2015 at 21:04, Lennart Regebro rege...@gmail.com wrote:
OK, so I realized another thing today, and that is that arithmetic
doesn't necessarily round trip.
For example, 2002-10-27 01:00 US/Eastern comes both in DST and STD.
But 2002-10-27 01:00 US/Eastern STD minus two days is 2002-10-25 01:00
US/Eastern DST
However, 2002-10-25 01:00 US/Eastern DST plus two days is 2002-10-27
01:00 US/Eastern, but it is ambiguous if you want DST or not DST.
And you can't pass in a is_dst flag to __add__, so the arithmatic must
just pick one, and the sensible one is to keep to the same DST.
import pytz
from datetime import datetime, timedelta
tz = pytz.timezone('US/Eastern')
dt = tz.localize(datetime(2002, 10, 27, 1, 0), is_dst=False)
dt2 = tz.normalize(dt - timedelta(days=2) + timedelta(days=2))
dt == dt2
True
tz.normalize(dt - timedelta(days=2))
datetime.datetime(2002, 10, 25, 2, 0, tzinfo=DstTzInfo 'US/Eastern'
EDT-1 day, 20:00:00 DST)
tz.normalize(tz.normalize(dt - timedelta(days=2)) + timedelta(days=2))
datetime.datetime(2002, 10, 27, 1, 0, tzinfo=DstTzInfo 'US/Eastern'
EST-1 day, 19:00:00 STD)
2002-10-27 01:00 US/Eastern is_dst=0 is after the DST transition
(EST). Subtracting 48 hours from it crosses the DST boundary and
should give you 2002-10-27 02:00 US/Eastern is_dst=1, prior to the DST
transition (EDT). Adding 48 hours again goes past 2002-10-27 01:00
EDT, crosses the DST boundary, and gives you back 2002-10-27 01:00
EST.
--
Stuart Bishop stu...@stuartbishop.net
http://www.stuartbishop.net/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
