[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: With Gregory's addition I think this issue can be considered fixed. Thank you Gregory. -- resolution: -> fixed stage: commit review -> resolved status: open -> closed ___ Python tracker

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Gregory P. Smith added the comment: Serhiy: should this one be marked fixed? -- assignee: -> serhiy.storchaka stage: patch review -> commit review ___ Python tracker ___

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Gregory P. Smith added the comment: It is fundamentally impossible for pickled data to magically cross the 2 and 3 language boundary unscathed. The basic str/bytes/unicode types in the language changed meaning. Code must be written manually by the data owners to fix that up based on what

[issue22005] datetime.__setstate__ fails decoding python2 pickle

miss-islington added the comment: New changeset 331bfa4f2c3026a35e111303df0f198d06b4e0c8 by Miss Islington (bot) in branch '3.7': bpo-22005: Document the reality of pickle compatibility. (GH-11054) https://github.com/python/cpython/commit/331bfa4f2c3026a35e111303df0f198d06b4e0c8 --

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Change by miss-islington : -- pull_requests: +10290 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Gregory P. Smith added the comment: New changeset e328753d91379274b699b93decff45de07854617 by Gregory P. Smith in branch 'master': bpo-22005: Document the reality of pickle compatibility. (GH-11054) https://github.com/python/cpython/commit/e328753d91379274b699b93decff45de07854617

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Change by Gregory P. Smith : -- pull_requests: +10289 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Christian Tanzer added the comment: Paul Ganssle wrote at Fri, 07 Dec 2018 17:22:36 +: > > Gregory P. Smith (gregory.p.smith) 2017-03-02 18:57 > > TL;DR - Just one more example of why nobody should *ever* use pickle > > under any circumstances. It is useless for data that is not

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Change by Paul Ganssle : -- nosy: -p-ganssle ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Paul Ganssle added the comment: I do not care enough about this to fight about it. The issue has been open long enough that I do not think it justified the urgency of rushing through a patch just before the release and merging without review, but now that it is in the release of multiple

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: This issue is already open for a long time. There is a problem which can not be worked around from the user side. I want to get it solved in 3.6, and today is the last chance for this. This is important for migrating from Python 2 to Python 3. You can

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Paul Ganssle added the comment: @Serhiy Any chance we can roll these back before the release so that they can have some time for discussion? I have serious concerns about having to support some Python 2/3 compatibility hack in datetime for the next 6 years. If this is worth doing at all, I

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: This is the same hack as in NumPy, so we are at least consistent here. I think we have to keep it some time after 2020, maybe to 2025. -- ___ Python tracker

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Paul Ganssle added the comment: I'm not sure I agree with how this was resolved. We're adding complexity to the datetime unpickler to support unpickling pickles created in Python 2 in Python 3? I also don't really understand the encoding parts of it, but it smells very fishy to me. I agree

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: New changeset 1133a8c0efabf6b33a169039cf6e2e03bfe6cfe3 by Serhiy Storchaka in branch 'master': bpo-22005: Fix condition for unpickling a date object. (GH-11025) https://github.com/python/cpython/commit/1133a8c0efabf6b33a169039cf6e2e03bfe6cfe3 --

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Change by Serhiy Storchaka : -- pull_requests: +10261 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: New changeset 19f6e83bf03b3ce22300638906bd90dd2dd5c463 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6': bpo-22005: Fixed unpickling instances of datetime classes pickled by Python 2. (GH-11017) (GH-11022) (GH-11024)

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Change by miss-islington : -- pull_requests: +10260 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: New changeset 0d5730e6437b157f4aeaf5d2e67abca23448c29a by Serhiy Storchaka in branch '3.7': [3.7] bpo-22005: Fixed unpickling instances of datetime classes pickled by Python 2. (GH-11017) (GH-11022)

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Change by Serhiy Storchaka : -- pull_requests: +10259 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: New changeset 8452ca15f41061c8a6297d7956df22ab476d4df4 by Serhiy Storchaka in branch 'master': bpo-22005: Fixed unpickling instances of datetime classes pickled by Python 2. (GH-11017)

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Change by Serhiy Storchaka : -- pull_requests: +10252 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: NumPy starves from the same issue. In NumPy this problem was solved by requiring encoding='latin1' passed to unpickler. It makes sense to use the same approach for datetime classes. -- versions: +Python 3.7, Python 3.8 -Python 2.7, Python 3.4,

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Change by Paul Ganssle : -- nosy: +p-ganssle ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Changes by Serhiy Storchaka : -- pull_requests: +699 ___ Python tracker ___ ___

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Gregory P. Smith added the comment: TL;DR - Just one more example of why nobody should *ever* use pickle under any circumstances. It is useless for data that is not transient for consumption by the same exact versions of all software that created it. Patches against 2.7 are not useful here.

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Changes by Torsten Bronger : -- nosy: +bronger ___ Python tracker ___ ___

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Christian Tanzer added the comment: This issue is getting old. Is there any way to solve this for Python 3.6? -- ___ Python tracker ___

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Christian Tanzer added the comment: IMNSHO, the problem lies in the Python 3 pickle.py and it is **not** restricted to datetime instances (for a detailed rambling see http://c-tanzer.at/en/breaking_py2_pickles_in_py3.html) . In Python 2, 8-bit strings are used for text and for binary data.

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Christian Tanzer added the comment: > The code works when using encoding='bytes'. Thanks Tim for the suggestion. > So this is not a bug, but is there any sense in having encoding='ASCII' by > default in pickle ? It is most definitely a bug. And it adds another road block to moving python

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: There are two issues here. 1. datetime.datetime accepts only bytes, not str. 2. Unpickling non-ASCII str pickled in Python 2 raises an error by default. The second issue usually hides the first one. The demonstration of the first issue: >>>

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Alexander Belopolsky added the comment: I wonder if this can be fixed using a fix_imports hook. I agree, it would be nice to fix this issue by modifying 3.x versions only. -- ___ Python tracker

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Alexander Belopolsky added the comment: > The first issue can be solved by accepting str argument and encoding it to > bytes. A strong -1 from me. Accepting bytes objects for year in 3.x (and str in 2.x) is a gross hack. In the long run, I would like to see a public named constructor, e.g.

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: The problem is that you can't unpickle a data that contains both datetime classes (datetime, date, time) instances and strings (including attribute names, so actually this affects instances of any Python classes). Yes, it only affects pickles transferred

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Alexander Belopolsky added the comment: > .. pickling and unpickling will be slower If we are concerned about performance, we should definitely avoid the decode-encode roundtrip. -- ___ Python tracker

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: Here is alternative patch for 2.7. It makes datetime pickling produce the same data as in 3.x. The side effect of this approach: it makes datetime pickling incompatible with Unicode disabled builds of 2.x. -- stage: -> patch review versions:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Serhiy Storchaka added the comment: Here is a patch against 2.7 that makes datetime pickling portable. It doesn't solve problem with existing pickled data, but at least it allows to convert existing pickled data with 2.7 to portable format. -- Added file:

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Edward O added the comment: The code works when using encoding='bytes'. Thanks Tim for the suggestion. So this is not a bug, but is there any sense in having encoding='ASCII' by default in pickle ? -- ___ Python tracker rep...@bugs.python.org

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Tim Peters added the comment: @eddygeek, I'd still call something so unintuitive a bug - it's hard to believe this is the _intended_ way to get it to work. So I'd keep this open until someone with better knowledge of intent chimes in. -- ___

[issue22005] datetime.__setstate__ fails decoding python2 pickle

New submission from Edward Oubrayrie: pickle.loads raises a TypeError when calling the datetime constructor, (then a UnicodeEncodeError in the load_reduce function). A short test program the log, including dis output of both PY2 and PY3 pickles, are available in this gist; and extract on

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Changes by Antoine Pitrou pit...@free.fr: -- nosy: +belopolsky, tim.peters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22005 ___ ___

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Changes by Alexander Belopolsky alexander.belopol...@gmail.com: -- nosy: +alexandre.vassalotti, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22005 ___

[issue22005] datetime.__setstate__ fails decoding python2 pickle

Tim Peters added the comment: I have no idea what was done to pickle for Python3, but this line works for me to unpickle a Python2 protocol 2 datetime pickle under Python3, where P2 is the Python2 pickle string: pickle.loads(bytes(P2, encoding='latin1'), encoding='bytes') For example,