[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

> Running the file couple of times will make the interpreter fail with: 
> libgcc_s.so.1 must be installed for pthread_cancel to work

See also bpo-44434: "_thread module: Remove redundant PyThread_exit_thread() 
call to avoid glibc fatal error: libgcc_s.so.1 must be installed for 
pthread_cancel to work".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[Yhojann Aguilera]

Yhojann Aguilera  added the comment:

I hope that when an error occurs, python tells me what the problem is. The 
abort core error is a problem at a lower level than python because python is 
not able to recognize or handle the error.

The main problem is that I exceeded the maximum number of process threads 
supported by the kernel.

What I hope is that python throws an exception when it exceeds this limit or 
when it cannot access the pointer in memory of the process thread.

The problem is not if the script is right or wrong, but that Python is not able 
to recognize and handle the problem. A generic message saying that an error 
occurred without indicating where and how it occurred is a python bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[Zack Weinberg]

Zack Weinberg  added the comment:

> To be clear: this issue is NOT a bug in Python

I don't think that's entirely true.  I think CPython needs to be linked against 
libgcc_s.so, so that this class of application bugs will no longer manifest as 
interpreter crashes.  I filed #37395 for that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

> Same problem using Python 3.6.8 on Ubuntu 18.04 LTS.

To be clear: this issue is NOT a bug in Python, but a bug in your application. 
You must fix your application. You can try to run it on Python 3.8 with 
python3.8 -X dev to get a log on the error. Good luck to debug it ;-)

The change that I made is only to add a log to help developers to find their 
own bugs ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[Yhojann Aguilera]

Yhojann Aguilera  added the comment:

Same problem using Python 3.6.8 on Ubuntu 18.04 LTS.

For now, solve this using

LD_PRELOAD=libgcc_s.so.1 python3 ...

For more details and pocs: https://github.com/WHK102/wss/issues/2

--
nosy: +Yhojann Aguilera
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

I closed the issue. I consider that the initial issue has been fixed and all 
related issues have been fixed as well:

* Python 3.8 now logs close() exception in file finalizer in development mode 
(-X dev) and debug mode (./configure --with-pydebug)
* I added sys.unraisablehook to Python 3.8 which allows to decide how these 
unraisable exceptions are handled
* Tests no longer log any "unraisable exception" in file finalizers (test_io 
and test_urllib have been fixed)
* regrtest now marks a test as ENV_CHANGED if it logs an unraisable exception 
(like the close() exception)

--

For Python 3.7, there is no known workaround.

It seems like Python 2.7 in debug mode logs an error:

$ python2.7-dbg bug.py 
close failed in file object destructor:
IOError: [Errno 9] Bad file descriptor

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.8, Python 3.9 -Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:


New changeset c15a682603a47f5aef5025f6a2e3babb699273d6 by Victor Stinner in 
branch '3.8':
bpo-37223: test_io: silence destructor errors (GH-14031)
https://github.com/python/cpython/commit/c15a682603a47f5aef5025f6a2e3babb699273d6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 4f6f7c5a611905fb6b81671547f268c226bc646a by Victor Stinner in 
branch 'master':
bpo-18748: Fix _pyio.IOBase destructor (closed case) (GH-13952)
https://github.com/python/cpython/commit/4f6f7c5a611905fb6b81671547f268c226bc646a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13821
pull_request: https://github.com/python/cpython/pull/13952

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -13234

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-37223: test_io logs Exception ignored in:  warnings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

Related issue: bpo-37054 "Ignored exceptions in test_memoryio".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:


New changeset bc2aa816620c5e02ad8e94d8514b7e8f3f551ca1 by Victor Stinner in 
branch 'master':
bpo-18748: _pyio.IOBase emits unraisable exception (GH-13512)
https://github.com/python/cpython/commit/bc2aa816620c5e02ad8e94d8514b7e8f3f551ca1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13429

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
pull_requests: +13234

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
pull_requests:  -13228

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Okay, opened issue36918 . Raising PR against that issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

Karthikeyan Singaravelan: Would you mind to open a separated issue for 
test_urllib warnings?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I was wrong so removing the close call when io_refs gets to zero causes error 
on test_urllib2 [0] where close is asserted to be True and I only tested 
test_urllib initially. One another way would be to check for fp.closed in 
http.client before flushing but as mentioned before I am not sure of the 
attribute being always present and also to modify http.client code just for 
tests.

[0] 
https://github.com/python/cpython/blob/f12ba7cd0a7370631214ac0b337ab5455ce717b2/Lib/test/test_urllib2.py#L1685

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
pull_requests: +13228

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

The ValueError warnings in test_urllib noted in msg340059  feels like an issue 
with the test where FakeSocket calls close by itself once it's internal io_refs 
counter is 0 and during destructor again close is called on an already closed 
object causing the ValueError.

* The test uses fakehttp() creating FakeSocket starting with io_refs as 1 [1]
* During the test in urlopen, sock.makefile() (io_refs += 1) is called 
increasing the io_refs to be 2.
* HTTPConnection.close calls sock.close (fakesocket.close) [3] calling io_refs 
-= 1 and to fakesocket.io_ref to be 1.
* This test uses raises status 302 with http_error_302 which calls 
self.redirect_internal where self.close is again called causing fp.io_refs == 0 
and subsequently calling io.BytesIO.close(self) to close the object. [4]
* During the end of the test destructor is called on the above closed 
fakesocket object and trying to flush on a closed object causes the ValueError 
warnings .

Maybe a check could be added during flush to make sure the object is not closed 
by testing for self.closed but I am not sure if closed attribute is guaranteed 
to be present. Removing the manual call to close in fakesocket could help since 
the destructor should be taking care of it and I could see no test failures or 
warnings removing the close as io_refs gets to 0.

[1] 
https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/test/test_urllib.py#L61
[2] 
https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/test/test_urllib.py#L67
[3] 
https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/http/client.py#L919
[4] 
https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/urllib/request.py#L2145

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 472f794a33221ea835a2fbf6c9f12aa2bd66d1b0 by Victor Stinner in 
branch 'master':
bpo-18748: test_io: silence destructor errors (GH-12805)
https://github.com/python/cpython/commit/472f794a33221ea835a2fbf6c9f12aa2bd66d1b0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

Maybe the fact that close() exceptions are ignored silently in io.IOBase 
constructor should be better documented, but I'm not sure where it should be 
documented. If someone has an idea, please go ahead and write a pull request :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

I wrote PR 12805 to silence IOBase destructor exceptions in test_io.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

Is there someone interested to debug remaining "Exception ignored:" logs in 
test_urllib?

test_invalid_redirect (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
ok

test_read_bogus (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
ok

test_redirect_limit_independent (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
ok


It's unclear to be if it's a bug in http.client, a bug in the test... or 
something else.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:

To debug remaining "Exception ignored in:" issues, I'm using the following 
patch:

diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py
index 972a4658b1..be38af3daa 100644
--- a/Lib/unittest/case.py
+++ b/Lib/unittest/case.py
@@ -710,6 +710,7 @@ class TestCase(object):
 
 # clear the outcome, no more needed
 self._outcome = None
+import gc; gc.collect()
 
 def doCleanups(self):
 """Execute all cleanup functions. Normally called for you after

And I run:

./python -X dev -u -m test test_io -v 2>&1|tee log

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12732

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 44235041f3b957abd36d3792450c3540aa09e120 by Victor Stinner in 
branch 'master':
bpo-18748: io.IOBase destructor now logs close() errors in dev mode (GH-12786)
https://github.com/python/cpython/commit/44235041f3b957abd36d3792450c3540aa09e120


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] io.IOBase destructor silence I/O error on close() by default

[STINNER Victor]

Change by STINNER Victor :


--
title: libgcc_s.so.1 must be installed for pthread_cancel to work -> io.IOBase 
destructor silence I/O error on close() by default

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com