[issue26571] turtle regression in 3.5

[Ellison Marks]

New submission from Ellison Marks:

I noticed some odd behaviour when running some turtle code I wrote using python 
3.5. A simplified example:

>>> from turtle import Turtle
>>> t = Turtle()
>>> t.getscreen().bye() # or manually close the turtle window
>>> t2 = Turtle()
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib64/python3.5/turtle.py", line 3816, in __init__
visible=visible)
  File "/usr/lib64/python3.5/turtle.py", line 2557, in __init__
self._update()
  File "/usr/lib64/python3.5/turtle.py", line 2660, in _update
self._update_data()
  File "/usr/lib64/python3.5/turtle.py", line 2646, in _update_data
self.screen._incrementudc()
  File "/usr/lib64/python3.5/turtle.py", line 1292, in _incrementudc
raise Terminator
turtle.Terminator
>>>

This code works under 3.4, opening a new turtle window the second time Turtle() 
is called. Under 3.5, a blank white window opens.

This seems to be related to https://hg.python.org/cpython/rev/1628484c9408, as 
the only point that raises Terminator is guarded by a check for `if not 
TurtleScreen._RUNNING:`

--
components: Library (Lib)
messages: 261840
nosy: Ellison Marks
priority: normal
severity: normal
status: open
title: turtle regression in 3.5
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19914] help([object]) returns "Not enough memory." on standard Python types, object and object functions

[Eryk Sun]

Eryk Sun added the comment:

For posterity for anyone that finds this old issue, I investigated this problem 
in the debugger in Windows 7. It turns out that more.com (the pager used by 
Python's help) calls MultiByteToWideChar [1] with dwFlags passed as 
MB_PRECOMPOSED (1), which is forbidden for UTF-8. The error message is just a 
generic error that incorrectly assumes decoding the byte string failed due to 
running out of memory.

You may be happy to learn that this problem is fixed in Windows 10.

[1]: https://msdn.microsoft.com/en-us/library/dd319072

Here are a few snapshots from the debugger.

more.com calls SetConsoleConversions from its init function, InitializeThings:

Breakpoint 0 hit
more!MORE::InitializeThings:
`ff293058 48895c2408  mov qword ptr [rsp+8],rbx
ss:`0024f7a0=
0:000> g
Breakpoint 2 hit
ulib!WSTRING::SetConsoleConversions:
07fe`f6498934 8a05d6a8mov al,byte ptr
[ulib!WSTRING::_UseAnsiConversions
 (07fe`f64a3210)] ds:07f
e`f64a3210=00

This causes decoding byte strings to use the current console codepage instead 
of the system ANSI or OEM codepage. The intention here is to allow a user to 
correctly display a text file that's in a different encoding. The decoded text 
is written to the console as Unicode via WriteConsoleW.

Here is the bad call where dwFlags (register rdx) is passed as MB_PRECOMPOSED 
(1), which is invalid for codepage 65001 (register rcx).

0:000> g
Breakpoint 1 hit
KERNELBASE!MultiByteToWideChar:
07fe`fd191f00 fff3pushrbx
0:000> ? @rcx
Evaluate expression: 65001 = `fde9
0:000> r rdx
rdx=0001

In Windows 10 this argument is passed as 0, the correct value.

This problem occurs indirectly via a utility library named ulib.dll, which is 
used by Windows command-line utilities. It should only occur when console 
conversions are enabled. Otherwise ulib converts using the system OEM and ANSI 
codepages.  I searched for other utilities that use 
ulib!WSTRING::SetConsoleConversions:

C:\>for %f in (C:\Windows\system32\*.exe) do @(^
More? dumpbin /imports "%f" | ^
More? findstr SetConsoleConversions && echo %f)
   7FF713B8934   167 ?SetConsoleConversions@WSTRING@@SAXXZ
C:\Windows\system32\find.exe

I found that find.exe is also subject to this bug in Windows 7. It fails to 
print the result if the console is using codepage 65001:

C:\Temp\test>type test
eggs
spam

C:\Temp\test>find /n "spam" *

-- TEST
[2]spam

C:\Temp\test>chcp 65001
Active code page: 65001

C:\Temp\test>find /n "spam" *

-- TEST

This works correctly in Windows 10.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26570] comma-separated cookies with expires header do not parse properly

[Ilya Kreymer]

New submission from Ilya Kreymer:

This is a peculiar regression in 3.5 where a comma-separated cookie following 
an expires field is considered invalid:

Ex: which results in a silent error/empty cookie in 3.5.1

Python 3.5.1 (default, Dec 26 2015, 18:11:22) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from http.cookies import SimpleCookie
>>> SimpleCookie('A=B; expires=Thu, 01-Jan-1970 00:00:00 GMT, C=D')


compared to 3.4.2 and 2.7.9

Python 3.4.2 (default, Jan 29 2015, 06:45:30) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from http.cookies import SimpleCookie
>>> SimpleCookie('A=B; expires=Thu, 01-Jan-1970 00:00:00 GMT, C=D')


Python 2.7.9 (default, Jan 29 2015, 06:28:46) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from Cookie import SimpleCookie
>>> SimpleCookie('A=B; expires=Thu, 01-Jan-1970 00:00:00 GMT, C=D')





This seems to specifically happen when the expires field is the last field in 
the cookie.
eg, this works as expected:

>>> SimpleCookie('A=B; expires=Thu, 01-Jan-1970 00:00:00 GMT; Path=/, C=D')


I'm not sure if this is related to the other comma-related cookie bugs, such as 
issue26302 as this does not appear to use the same regex at first glance. I 
have not had a chance to track it down it further.

--
components: Library (Lib)
messages: 261839
nosy: ikreymer
priority: normal
severity: normal
status: open
title: comma-separated cookies with expires header do not parse properly
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18320] python installation is broken if prefix is overridden on an install target

[Ned Deily]

Ned Deily added the comment:

Allowing prefix to be overridden after configure-time is certainly not the most 
critical shortcoming of the Python build system and it is possible someone is 
relying on that behavior.  Since no one, including me, seems very interested at 
this point in trying to change it, let's close this issue.

--
resolution:  -> wont fix
stage: needs patch -> resolved
status: open -> closed
versions: +Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26562] Large Involuntary context switches during oom-killer

[Damian Myerscough]

Damian Myerscough added the comment:

Thanks for the feedback, I will continue to dig into this. 

I know processes go crazy sometimes when OOM killer kicks off, I just wanted to 
rule out a Python bug or if anyone in the community has seen this before.

Thanks

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18320] python installation is broken if prefix is overridden on an install target

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25597] unittest.mock does not wrap dunder methods (__getitem__ etc)

[Robert Collins]

Robert Collins added the comment:

I think this is a valid mock bug; it likely needs some thoughtful exhaustive 
testing, and obviously support added for it.

--
stage:  -> test needed
title: unittest.mock does not wrap dict objects correctly -> unittest.mock does 
not wrap dunder methods (__getitem__ etc)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25894] unittest subTest failure causes result to be omitted from listing

[Robert Collins]

Robert Collins added the comment:

The basic model is this:
 - a test can have a single outcome [yes, the api is ambiguous, but there it is]
 - subtests let you identify multiple variations of a single test (note the id 
tweak etc) and *may* be reported differently

We certainly must not report the test as a whole passing if any subtest did not 
pass.

Long term I want to remove the error/failure partitioning of exceptions; its 
not actually useful.

The summary for the test, when subtests are used, should probably enumerate the 
states.

test_foo (3 passed, 2 skipped, 1 failure)

in much the same way the run as a whole is enumerated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26295] Random failures when running test suite in parallel (-m test -j0) caused by test_regrtest

[STINNER Victor]

STINNER Victor added the comment:

Hum, test_doctest, test_zipfile and test_pyclbr fail if test is converted to a 
package (if Lib/test/__init__.py is removed).

Attached patch fixes test_zipfile.

I created the issue #26569 for test_pyclbr failure.

--
Added file: http://bugs.python.org/file42177/test_regrtest_tmpdir-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26569] pyclbr.readmodule() and pyclbr.readmodule_ex() doens't support packages

[STINNER Victor]

New submission from STINNER Victor:

While working on the issue #26295 which converts the test module to a package, 
I noticed that pyclbr doesn't work with packages: test_pyclbr fails when test 
becomes a package.

Attached patch fixes pyclbr._readmodule():

* Replace "spec.loader.is_package(fullmodule)" test with 
"spec.submodule_search_locations is not None" to check is the module is a 
package
* for a package, use spec.submodule_search_locations to build __path__

I don't know importlib well enough to say if my usage of importlib spec is 
correct.

--
files: pyclbr.patch
keywords: patch
messages: 261832
nosy: brett.cannon, haypo
priority: normal
severity: normal
status: open
title: pyclbr.readmodule() and pyclbr.readmodule_ex() doens't support packages
versions: Python 3.6
Added file: http://bugs.python.org/file42176/pyclbr.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26569] pyclbr.readmodule() and pyclbr.readmodule_ex() don't support packages

[STINNER Victor]

Changes by STINNER Victor :


--
title: pyclbr.readmodule() and pyclbr.readmodule_ex() doens't support packages 
-> pyclbr.readmodule() and pyclbr.readmodule_ex() don't support packages

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26564] Malloc debug hooks: display memory block traceback on error

[Roundup Robot]

Roundup Robot added the comment:

New changeset 769dfcb701ee by Victor Stinner in branch 'default':
Issue #26564: Fix test_capi
https://hg.python.org/cpython/rev/769dfcb701ee

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26538] regrtest: setup_tests() must not replace module.__path__ (_NamespacePath) with a simple list // importlib & abspath

[Roundup Robot]

Roundup Robot added the comment:

New changeset 96b73b649b15 by Victor Stinner in branch 'default':
regrtest: Fix module.__path__
https://hg.python.org/cpython/rev/96b73b649b15

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26564] Malloc debug hooks: display memory block traceback on error

[Roundup Robot]

Roundup Robot added the comment:

New changeset 8215dae7ec3c by Victor Stinner in branch 'default':
Oops, revert unwanted change used to create an example
https://hg.python.org/cpython/rev/8215dae7ec3c

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26564] Malloc debug hooks: display memory block traceback on error

[Roundup Robot]

Roundup Robot added the comment:

New changeset cef6a32d805f by Victor Stinner in branch 'default':
On memory error, dump the memory block traceback
https://hg.python.org/cpython/rev/cef6a32d805f

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26564] Malloc debug hooks: display memory block traceback on error

[Roundup Robot]

Roundup Robot added the comment:

New changeset 18a19e62bac5 by Victor Stinner in branch 'default':
Enhance and rewrite traceback dump C functions
https://hg.python.org/cpython/rev/18a19e62bac5

New changeset fea3c6e9a38e by Victor Stinner in branch '3.5':
_tracemalloc: store lineno as unsigned int
https://hg.python.org/cpython/rev/fea3c6e9a38e

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18320] python installation is broken if prefix is overridden on an install target

[Martin Panter]

Martin Panter added the comment:

I agree with Martin v. Löwis that this is not a real bug. Cosmicduck doesn’t 
seem aware of the “configure --prefix=. . .” option, but I don’t think it is 
right to add hacks to the makefile to prevent overriding variables.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Xavier de Gaye]

Xavier de Gaye added the comment:

> But I am a bit worried at the new makefile syntax [ifeq directive and 
> $(findstring) function]. I suspect it is Gnu specific, and that Python tries 
> to support other versions of Make as well.

Yes, they are both GNU extensions :(

To avoid modifying configure.ac since the information is already available, the 
recipes could be modified with the following conditional:

cross_compiling=$$(echo $(PYTHON_FOR_BUILD) | sed 
"s/.*_PYTHON_HOST_PLATFORM.*/yes/"); \
if [ "$$cross_compiling" = "yes" ]; then \
  touch $@;
else \
  # The original recipes.
fi

So the built binaries $(PGEN) and Programs/_freeze_importlib would be dummy 
empty files, but that would not prevent Programs/_freeze_importlib.o, 
$(LIBRARY_OBJS_OMIT_FROZEN) and $(PGENOBJS) to be needlessly cross-compiled.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26568] Add a new warnings.showmsg() function taking a warnings.WarningMessage object

[STINNER Victor]

STINNER Victor added the comment:

I'm not sure about the method names showmsg() and formatmsg().

Maybe: showwarnmsg() and formatwarnmsg()?

First I used warnings.show() and warnings.format(), but "format()" is already 
the name of a builtin function.

By the way, I would prefer to use underscore between names, like show_msg() or 
show_warn_msg(). But I don't know if it's ok to use a different naming scheme 
in an existing module :-/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24263] unittest cannot load module whose name starts with Unicode

[Robert Collins]

Robert Collins added the comment:

sih4sing5hong5  - I think we do need a test in fact - it can be done using 
mocks, but right now I think the patch has a bug - it looks for isidentifier on 
$thing.py, but not on just $thing (which we need to do to handle packages, vs 
modules).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26560] Error in assertion in wsgiref.handlers.BaseHandler.start_response

[SilentGhost]

Changes by SilentGhost :


--
nosy: +pje
type:  -> behavior
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26565] [ctypes] Add value attribute to non basic pointers.

[SilentGhost]

Changes by SilentGhost :


--
nosy: +amaury.forgeotdarc, belopolsky, meador.inge
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20891] PyGILState_Ensure on non-Python thread causes fatal error

[STINNER Victor]

STINNER Victor added the comment:

Attached patch should fix the issue.

--
keywords: +patch
Added file: http://bugs.python.org/file42174/PyGILState_Ensure.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20891] PyGILState_Ensure on non-Python thread causes fatal error

[STINNER Victor]

STINNER Victor added the comment:

ptest.c: Portable example using Python API.

--
Added file: http://bugs.python.org/file42173/ptest.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26102] access violation in PyErrFetch if tcur==null in PyGILState_Release

[STINNER Victor]

STINNER Victor added the comment:

Good news: it looks like I fixed the Py_FatalError() bug in the issue #26558. 
Py_FatalError() can again be called with the GIL released.

The bug was introduced in Python 3.0.

Since your issue is a bug in your code and that Py_FatalError() was fixed, I 
close the issue. Thanks for your bug report.

--
nosy: +haypo
resolution:  -> fixed
status: open -> closed
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26558] Disable PyGILState_Check() when Py_NewInterpreter() is called and add more checks on the GIL

[Roundup Robot]

Roundup Robot added the comment:

New changeset b394fc71f92a by Victor Stinner in branch '3.5':
Fix Py_FatalError() if called without the GIL
https://hg.python.org/cpython/rev/b394fc71f92a

New changeset c298c6d8b324 by Victor Stinner in branch '3.5':
faulthandler: Test Py_FatalError() with GIL released
https://hg.python.org/cpython/rev/c298c6d8b324

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26558] Disable PyGILState_Check() when Py_NewInterpreter() is called and add more checks on the GIL

[STINNER Victor]

STINNER Victor added the comment:

Ok, it looks like buildbots are happy. I close the issue.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26564] Malloc debug hooks: display memory block traceback on error

[STINNER Victor]

STINNER Victor added the comment:

I reviewed my own patch on Rietveld :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26568] Add a new warnings.showmsg() function taking a warnings.WarningMessage object

[STINNER Victor]

New submission from STINNER Victor:

Currently, the warnings.showformat() function take between 4 and 6 parameters 
and it's not possible to add new parameters.

warnings.showformat() calls warnings.formatwarnings() with 5 parameters. Again, 
it's not easy to pass new parameters.

I would like to add a new "source" parameter for ResourceWarning: see issue 
#26567 "ResourceWarning: Use tracemalloc to display the traceback where an 
object was allocated when a ResourceWarning is emitted".

To make warnings extensible, I propose to:

* Add new showmsg() and formatmsg() functions to the warnings module, these 
functions take a warnings.WarningMessage instance
* The _warnings module calls warnings.showmsg() rather than 
warnings.showwarning()
* For backward compatibility, warnings.showmsg() calls warnings.showwarning() 
if warnings.showwarning() was replaced. Same for warnings.formatmsg(): call 
warnings.formatwarning() if replaced.

Attached patch implements these changes.

With these changes, the warnings looks a little bit more like the logging 
module and its logging.LogRecord class.

The patch is incomplete, it doesn't add much tests yet and it doesn't touch the 
documentation. I would prefer to get the new API (and the whole idea) approved 
before going too far.

Note: If warnings.showwarning is deleted ("del warnings.showwarning"), 
showmsg() uses its own implementation. Same for formatmsg().

--
files: warnings_showmsg.patch
keywords: patch
messages: 261814
nosy: haypo
priority: normal
severity: normal
status: open
title: Add a new warnings.showmsg() function taking a warnings.WarningMessage 
object
versions: Python 3.6
Added file: http://bugs.python.org/file42172/warnings_showmsg.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26567] ResourceWarning: Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted

[STINNER Victor]

STINNER Victor added the comment:

> Backward-compatibility problem: The C function PyErr_ResourceWarning() always 
> call warnings.showwarning() with the keyword parameter source. If an 
> application replaces the warnings.showwarning() function, it will probably 
> fail because it doesn't know the source parameter.

I proposed the issue #26568 "Add a new warnings.showmsg() function taking a 
warnings.WarningMessage object" to fix this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26040] Improve coverage and rigour of test.test_math

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +christian.heimes, eric.smith, lemburg, serhiy.storchaka, stutzbach
versions: +Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26564] Malloc debug hooks: display memory block traceback on error

[STINNER Victor]

STINNER Victor added the comment:

See also the issue #26567: "Use tracemalloc to display the traceback where an 
object was allocated when a ResourceWarning is emitted".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26567] ResourceWarning: Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted

[STINNER Victor]

Changes by STINNER Victor :


--
title: Use tracemalloc to display the traceback where an object was allocated 
when a ResourceWarning is emitted -> ResourceWarning: Use tracemalloc to 
display the traceback where an object was allocated when a ResourceWarning is 
emitted

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26567] Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted

[STINNER Victor]

New submission from STINNER Victor:

Python emits ResourceWarning when an object using limited resource is destroyed 
without being explicitly closed: files, sockets, etc. The problem is that it's 
hard to find where the object comes from, since the warning can occur very 
late, in the garbage collector, etc.

I propose to reuse tracemalloc.get_object_traceback() to show were the object 
was allocated, when tracemalloc traces memory allocations. In practice, I 
propose to add a new "source" parameter to warnings.showwarning().

Attached patch:

* Add a new keyword-only source parameter to warnings.showwarning()
* Add C function PyErr_ResourceWarning() to pass source
* showwarning() uses tracemalloc.get_object_traceback() to get the traceback 
were the object was allocated
* Modify socket.socket, io.FileIO and os.scandir destructor to use 
PyErr_ResourceWarning()


Backward-compatibility problem: The C function PyErr_ResourceWarning() always 
call warnings.showwarning() with the keyword parameter source. If an 
application replaces the warnings.showwarning() function, it will probably fail 
because it doesn't know the source parameter.

I don't know how to handle this backward compatibility issue.


The patch is incomplete, it's not possible yet to emit a warning in pure Python 
with a source parameter.


x.py script used for examples below:
-
import warnings
import os
import socket

def func2():
#f=open("/etc/issue")
#f=os.scandir('.')
f=socket.socket()
f=None

def func():
func2()

func()
-


Output with Python 3.5:
-
x.py:9: ResourceWarning: unclosed 
  f=None
-


Output with -X tracemalloc=5 command line option and patched Python 3.6:
-
x.py:9: ResourceWarning: unclosed 
  f=None
Object allocated at (most recent call first):
  File "x.py", lineno 8
f=socket.socket()
  File "x.py", lineno 12
func2()
  File "x.py", lineno 14
func()
-

It's much easier to understand where the warning comes from, no? At x.py:8, 
line "f=socket.socket()".

Note: the traceback doesn't contain the function name, since tracemalloc only 
stores filename and line number.


See also the issue #26564 "Malloc debug hooks: display memory block traceback 
on error".


For Python < 3.6, I wrote "res_warn.py" script which monkey-patches io.FileIO 
and socket.socket to implement something similar. The script is a fragile hack. 
I would prefer to have the feature built-in Python.

https://bitbucket.org/haypo/misc/src/0a40f27360424145bad0f9b62c9e9148ffdbb169/python/res_warn.py

--
files: warnings_tracemalloc.patch
keywords: patch
messages: 261812
nosy: haypo, serhiy.storchaka, yselivanov
priority: normal
severity: normal
status: open
title: Use tracemalloc to display the traceback where an object was allocated 
when a ResourceWarning is emitted
versions: Python 3.6
Added file: http://bugs.python.org/file42171/warnings_tracemalloc.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24230] tempfile.mkdtemp() doesn't work with bytes paths

[STINNER Victor]

STINNER Victor added the comment:

"""
Today we're doing something like this:

tmpdir = tempfile.mkdtemp('', 'hgtests.', d and 
d.decode('utf-8')).encode('utf-8')
"""

Don't do that. UTF-8 is not the right encoding.

Use os.fsencode() and os.fsdecode(). Internally, Python uses 
sys.getfilesystemencoding() with 'surrogateescape' error handler (but 'strict' 
error handler on Windows).

If you use UTF-8, you can get indirectly mojibake.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24230] tempfile.mkdtemp() doesn't work with bytes paths

[STINNER Victor]

STINNER Victor added the comment:

I don't understand the rationale to add new functions using bytes filenames.

IMHO it's a bad practice to pass bytes filenames to write portable code. On 
Windows, bytes lead to strange bugs: os.listdir(bytes) gives you filenames 
which don't exist when a filename cannot be encoded to the ANSI code page.

On UNIX, os.fsencode() and os.fsdecode() allows you to convert filenames 
between unicode and bytes. It provides nice properties like 
os.fsencode(os.fsdecode(value))==value where value is bytes. No need to add new 
APIs, use existing APIs an use os.fsencode() and os.fsdecode() on top of them.

Bytes filenames are deprecated on Windows since Python 3.2. The new 
os.scandir() function doesn't support bytes for example, it's a deliberate 
choice.

Maybe I missed an important technical issue?

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26562] Large Involuntary context switches during oom-killer

[STINNER Victor]

STINNER Victor added the comment:

> When oom-killer is trigger

IMHO this is your root problem.

> When oom-killer is trigger I see a large number of involuntary context 
> switches ...

The principle of *involuntarty* context switches is that the application is not 
responsible for them.

> I tried the same test using NodeJS/C++ and I could see a lot less involuntary 
> context switches and major page faults which indicates this could be a Python 
> issue.

Your rationale cannot be good.

It is not a Python bug.

OOM Killer is not a bug but a feature to limit abuse of resources on Linux.

But yeah, in my experience, Linux becomes crazy when it reachs OOM Killer. You 
can try to tune your kernel, but it's much better to always avoid using all 
memory ;-)

--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26566] Failures on FreeBSD CURRENT buildbot

[Roundup Robot]

Roundup Robot added the comment:

New changeset 93b998d47bc0 by Victor Stinner in branch 'default':
Issue #26566: Rewrite test_signal.InterProcessSignalTests
https://hg.python.org/cpython/rev/93b998d47bc0

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26523] multiprocessing ThreadPool is untested

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
resolution:  -> fixed
stage: commit review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26523] multiprocessing ThreadPool is untested

[Roundup Robot]

Roundup Robot added the comment:

New changeset 4cabf12b7fc6 by Antoine Pitrou in branch '3.5':
Issue #26523: The multiprocessing thread pool (multiprocessing.dummy.Pool) was 
untested.
https://hg.python.org/cpython/rev/4cabf12b7fc6

New changeset 131f92510164 by Antoine Pitrou in branch 'default':
Issue #26523: The multiprocessing thread pool (multiprocessing.dummy.Pool) was 
untested.
https://hg.python.org/cpython/rev/131f92510164

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26562] Large Involuntary context switches during oom-killer

[Antoine Pitrou]

Antoine Pitrou added the comment:

Why would it be a Python issue? By construction I doubt the OOM killer lets the 
process run anything before force-killing it.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26566] Failures on FreeBSD CURRENT buildbot

[koobs]

Changes by koobs :


--
nosy: +koobs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26566] Failures on FreeBSD CURRENT buildbot

[STINNER Victor]

New submission from STINNER Victor:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%203.x/builds/218/steps/test/logs/stdio

==
FAIL: test_terminate (test.test_subprocess.POSIXProcessTestCase)
--
Traceback (most recent call last):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_subprocess.py",
 line 1679, in test_terminate
self.assertEqual(p.wait(), -signal.SIGTERM)
AssertionError: 0 != -15

--


==
FAIL: test_main (test.test_signal.InterProcessSignalTests)
--
Traceback (most recent call last):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_signal.py",
 line 197, in test_main
self.fail(tb)
AssertionError: Traceback (most recent call last):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_signal.py",
 line 180, in test_main
self.run_test()
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_signal.py",
 line 117, in run_test
self.assertTrue(self.a_called)
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/case.py",
 line 677, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true



[ 94/400/2] test_multiprocessing_main_handling
Timeout (1:00:00)!
Thread 0x000801815000 (most recent call first):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", 
line 376 in select
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", 
line 1709 in _communicate
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", 
line 1075 in communicate
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/support/script_helper.py",
 line 86 in run_python_until_end
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/support/script_helper.py",
 line 96 in _assert_python
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/support/script_helper.py",
 line 135 in assert_python_ok
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_multiprocessing_main_handling.py",
 line 155 in _check_script
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_multiprocessing_main_handling.py",
 line 182 in test_ipython_workaround
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/case.py",
 line 600 in run
(...)



http://buildbot.python.org/all/builders/AMD64%20FreeBSD%20CURRENT%203.x/builds/217/steps/test/logs/stdio


[  6/400] test_multiprocessing_fork
Timeout (1:00:00)!
Thread 0x000801815000 (most recent call first):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/connection.py",
 line 379 in _recv
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/connection.py",
 line 407 in _recv_bytes
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/connection.py",
 line 250 in recv
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/managers.py",
 line 717 in _callmethod
  File "", line 2 in terminate
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py",
 line 1837 in test_terminate
(...)

--
keywords: buildbot
messages: 261805
nosy: haypo
priority: normal
severity: normal
status: open
title: Failures on FreeBSD CURRENT buildbot
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26523] multiprocessing ThreadPool is untested

[Antoine Pitrou]

Antoine Pitrou added the comment:

> Is it worth to change multiprocessing.Pool in _TestPoolWorkerErrors
> and _TestPoolWorkerLifetime? Currently they work only for processes,
> but may be they can be generalized.

Perhaps as another issue?  It may be more involved than this one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17758] test_site fails when the user does not have a home directory

[STINNER Victor]

STINNER Victor added the comment:

Everything is possible. It's just a matter of priorities. Do you want to
work on a patch. I'm not really interested. If you consider that it'd
worth, open a new issue.

I close this issue since the iniyal bug is closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17758] test_site fails when the user does not have a home directory

[koobs]

koobs added the comment:

@Victor, it would be nice if this test could instead use/fake/mock what it 
needs in the build directory, or is this not possible because it needs an 
'actual' user/group context to work?

--
nosy: +koobs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26499] http.client.IncompleteRead from HTTPResponse read after part reading file

[Martin Panter]

Martin Panter added the comment:

Here is a modified version of the tests that I propose to commit soon. I 
trimmed the size of the tests down and simplified some of them. I also 
mentioned that BufferedIOBase is supported in the documentation.

--
stage: patch review -> commit review
Added file: http://bugs.python.org/file42170/issue26499_6.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com