[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
Removed message: https://bugs.python.org/msg321110

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
Removed message: https://bugs.python.org/msg32

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Paul Ganssle]

Paul Ganssle  added the comment:

Oops, that previous comment was intended for a completely different ticket. My 
mistake.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Paul Ganssle]

Paul Ganssle  added the comment:

Hmm. I never noticed this. In the past I have used the (undocumented) 
PyDateTimeAPI struct, which the official macros wrap. I'm not sure how official 
that struct is considering it doesn't show up in the documentation.

I agree that it should be possible to construct TZ-aware datetimes using the 
official C API, so I guess the question is whether to add a bunch more macros 
or document the existing struct.

--
nosy: +p-ganssle

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Frank Blankenburg]

Changes by Frank Blankenburg :


--
nosy: +FrankBlabu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Nathan Jensen]

Changes by Nathan Jensen :


--
nosy: +Nathan Jensen

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Nick Coghlan]

Nick Coghlan added the comment:

I've added Steve Dower to the nosy list, as he's done some work recently on 
making the Windows builds more embedding-friendly, and I believe at least some 
of that time may have been funded work.

Unfortunately, we don't currently have anyone I'm aware of that's being 
specifically paid to improve or maintain CPython's embedding support in 
general, so getting attention for these kinds of interpreter reinitialization 
bugs can be a bit hit-or-miss.

--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Christopher Schramm]

Christopher Schramm added the comment:

This issue should have a much higher priority as it basically breaks Python 
embedding unless the user either does not re-initialize the interpreter or 
avoid the use of _datetime.strptime.

We're currently testing with a patch based on Christian and Alexander's idea 
but using m_free as using m_clear and Py_CLEAR neither makes sense to me nor 
did it work in conjunction with Py_Finalize when testing it. A version matching 
the current tip is attached. We did not run into any issues so far (and the 
only thing I can think of is clearing the static variable too often and thus 
causing some extra imports).

--
Added file: http://bugs.python.org/file46478/27400.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Christopher Schramm]

Changes by Christopher Schramm :


--
nosy: +cschramm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Denny Weinberg]

Denny Weinberg added the comment:

Any news here? 3.6.0 is also affected by this bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Chi Hsuan Yen]

Changes by Chi Hsuan Yen :


--
nosy: +Chi Hsuan Yen

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Jim Jewett]

Jim Jewett added the comment:

Having to (re-)fill the cache once per interpreter seems like a reasonable 
price to pay.

Why is 3.5 not included?  Did this not cause problems before the import change, 
or is it just that this bug is small enough that maybe it isn't worth 
backporting?

--
nosy: +Jim.Jewett

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Christian Heimes]

Christian Heimes added the comment:

Wouldn't it clear strptime_module when a subinterpreter shuts down, too? It's 
not a big deal because it can't cause a crash.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

Yes, I think something like the attached patch may do the trick.

--
assignee:  -> belopolsky
keywords: +patch
versions: +Python 3.6, Python 3.7 -Python 3.5
Added file: http://bugs.python.org/file44678/issue27400.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Christian Heimes]

Christian Heimes added the comment:

PEP 3121 is a big change. Can we use PyModuleDef->m_clear() for a clever hack?

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

I am not sure this is possible to fix without refactoring the datetime module 
according to PEP 3121. See #15390.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Josh Rosenberg]

Josh Rosenberg added the comment:

Hmm... On checking down some of the code paths and realizing there were some 
issues in 3.5 (redundant code, and what looked like two memory leaks), I 
checked tip (to avoid opening bugs on stale code), and discovered that #22557 
rewrote the import code, reducing the cost of top level reimport by ~60%, so my 
microbenchmarks (run on Python 3.5.0) are already out of date for 3.6's faster 
re-import. Even so, caching wasn't a wholly unreasonable optimization before 
now, and undoing it now still has a cost, if a smaller one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Josh Rosenberg]

Josh Rosenberg added the comment:

Nick: Looks like it's quite a bit more work than just a dict lookup. That 
PyImport_ImportModuleNoBlock call (which seems odd; the implementation of 
NoBlock is just to wrap the blocking function; guess we don't allow 
non-blocking imports anymore and this is just to avoid changing all the names 
elsewhere?) involves a *lot* more work than just a dict lookup (it devolves to 
a PyImport_Import call 
https://hg.python.org/cpython/file/3.5/Python/import.c#l1743 , which basically 
does everything involved in the import process aside from actually 
reading/parsing the file unconditionally, because of how weird __import__ 
overrides can be, I guess).

While it's not a perfect comparison, compare:

>>> import _strptime  # It's now cached

# Cache globals dict for fair comparison without globals() call overhead
>>> g = globals() 

# Reimport (this might be *more* expensive at C layer, see notes below)
>>> %timeit -r5 import _strptime
100 loops, best of 5: 351 ns per loop

# Dict lookup (should be at least a bit cheaper at C layer if done 
equivalently, using GetAttrId to avoid temporary str)
>>> %timeit -r5 g['_strptime']
1000 loops, best of 5: 33.1 ns per loop

# Cached reference (should be *much* cheaper at C layer)
>>> %timeit -r5 _strptime
1 loops, best of 5: 19.1 ns per loop

Note: I'm a little unclear on whether a Python function implemented in C has 
its own globals, or whether it's simulated as part of the C module 
initialization); if it lacks globals, then the work done for PyImport_Import 
looks like it roughly doubles (it has to do all sorts of work to simulate 
globals and the like), so that 351 ns per re-import might actually be costlier 
in C.

Either way, it's a >10x increase in cost to reimport compared to a dict lookup, 
and ~18x speedup over using a cached reference (and like I said, I think the 
real cost of the cheaper options would be much less in C, so the multiplier is 
higher). Admittedly, in tests, empty string calls to `_strptime._strptime` take 
around 7.4 microseconds (with realistic calls taking 8.5-13.5 microseconds), so 
caching is saving maybe a third of a microsecond overhead, maybe 2.5%-4.5% of 
the work involved in the strptime call.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Roman Evstifeev]

Changes by Roman Evstifeev :


--
nosy: +Roman.Evstifeev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Nick Coghlan]

Nick Coghlan added the comment:

Aye, skipping the caching entirely would be an even simpler solution - the only 
thing it is saving in the typical case is a dictionary lookup in the modules 
cache.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Ammar Askar]

Ammar Askar added the comment:

Is there any particular reason that datetime.strptime caches the imported 
module like that? 

>From a quick search, these two other examples don't bother with any caching: 

https://github.com/python/cpython/blob/2d264235f6e066611b412f7c2e1603866e0f7f1b/Modules/timemodule.c#L709

https://github.com/python/cpython/blob/64fe35c9fee088f7fec4dd2d760cb0026ac54ec8/Python/traceback.c#L277

--
nosy: +ammar2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Nick Coghlan]

Nick Coghlan added the comment:

Thanks for the report Denny. Looking at 
https://hg.python.org/cpython/file/30099abdb3a4/Modules/datetimemodule.c#l3929, 
there's a problematic caching of the "_strptime" module that is almost 
certainly the cause of the problem - it will attempt to call 
_strptime._strptime from the already finalized interpreter rather than the new 
one.

It should be possible to adjust that logic to permit a check for 
_strptime._strptime being set to None, and reimporting _strptime in that case.

--
nosy: +belopolsky, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Denny Weinberg]

Denny Weinberg added the comment:

Just to be clear:

The error happens after these steps:

1. Call strptime
2. Call cpython function "Py_Finalize" and "Py_Initialize"
3. Call strptime again

Now we get the error "attribute of type 'NoneType' is not callable"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize

[Denny Weinberg]

New submission from Denny Weinberg:

After calling Py_Finalize and Py_Initialize I get the message "attribute of 
type 'NoneType' is not callable" on the datetime.strptime method.

Example:
from datetime import datetime
s = '20160505 16'
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')

The first call works fine but it crashes after the re initialization.

Workaround:
from datetime import datetime
s = '20160505 16'
try:
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')
except TypeError:
import time
refdatim = datetime.fromtimestamp(time.mktime(time.strptime(s, '%Y%m%d 
%H%M%S')))

Related Issue: Issue17408 ("second python execution fails when embedding")

--
components: Interpreter Core
messages: 269379
nosy: Denny Weinberg, palm.kevin
priority: normal
severity: normal
status: open
title: Datetime NoneType after calling Py_Finalize and Py_Initialize
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com