[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[STINNER Victor]

STINNER Victor  added the comment:

This bug has been fixed in Python 2.7.15.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Benjamin Peterson]

Change by Benjamin Peterson :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Benjamin Peterson]

Benjamin Peterson  added the comment:


New changeset dbf52e02f18dac6f5f0a64f78932f3dc6efc056b by Benjamin Peterson in 
branch '2.7':
bpo-31530: fix crash when multiple threads iterate over a file, round 2 (#5060)
https://github.com/python/cpython/commit/dbf52e02f18dac6f5f0a64f78932f3dc6efc056b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Benjamin Peterson]

Change by Benjamin Peterson :


--
pull_requests: +4936
stage: resolved -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Benjamin Peterson]

Benjamin Peterson  added the comment:

Unfortunately, it looks like this fix causes a regression. Some programs rely 
on being able to seek() and write to a file on multiple threads. For example, 
py.test captures the standard streams by redirecting them to a tmpfile and then 
truncating+seeking to 0 after every test. This change broke that situation when 
multiple threads are logging. Anyway, there's no real fundamental reason to 
prevent concurrent access, since the underlying stdio implementation is 
threadsafe. I think we'll have to resurrect my PR.

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 6401e5671781eb217ee1afb4603cc0d1b0367ae6 by Serhiy Storchaka in 
branch '2.7':
[2.7] bpo-31530: Stop crashes when iterating over a file on multiple threads. 
(#3672)
https://github.com/python/cpython/commit/6401e5671781eb217ee1afb4603cc0d1b0367ae6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[STINNER Victor]

STINNER Victor added the comment:

> Iteration on Python 2 files is fragile, now is not the time to fix 
> fundamental bugs in Python 2.

Right. I don't have a strong preference for Serhiy's PR 3672.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Guido van Rossum]

Guido van Rossum added the comment:

It should be like 'if key in dct: del dct[key]'. It may fail when two
threads do this but the *internal* state of dct should not be compromised.

On Sep 25, 2017 2:45 AM, "Antoine Pitrou"  wrote:

>
> Antoine Pitrou added the comment:
>
> Le 25/09/2017 à 11:39, STINNER Victor a écrit :
> >
> > Antoine: "I'm wary of raising an exception.  Assuming Benjamin's patch
> is correct, I vote for it."
> >
> > Can you please elaborate? You are fine with the fact that iter(file) in
> two thread can return the same line?
>
> AFAIU it already could.  Iteration on Python 2 files is fragile, now is
> not the time to fix fundamental bugs in Python 2.
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Antoine Pitrou]

Antoine Pitrou added the comment:

Le 25/09/2017 à 11:39, STINNER Victor a écrit :
> 
> Antoine: "I'm wary of raising an exception.  Assuming Benjamin's patch is 
> correct, I vote for it."
> 
> Can you please elaborate? You are fine with the fact that iter(file) in two 
> thread can return the same line?

AFAIU it already could.  Iteration on Python 2 files is fragile, now is
not the time to fix fundamental bugs in Python 2.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[STINNER Victor]

STINNER Victor added the comment:

Antoine: "I'm wary of raising an exception.  Assuming Benjamin's patch is 
correct, I vote for it."

Can you please elaborate? You are fine with the fact that iter(file) in two 
thread can return the same line?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Antoine Pitrou]

Antoine Pitrou added the comment:

I'm wary of raising an exception.  Assuming Benjamin's patch is correct, I vote 
for it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[STINNER Victor]

STINNER Victor added the comment:

It's difficult to make a choice between Benjamin's PR 3670 and Serhiy's PR 3672.

Benjamin wrote that about his PR:

> No attempt is made to define or provide "reasonable" semantics for iterating
> over a file on multiple threads. (Non-crashing) races are still
> present. Duplicated, corrupt, and missing data will happen.

I'm not confortable with the idea of data corruption. I'm not confortable with 
breaking backward compatibility neither. IMHO the key question is if anyone 
actually use a file object at the same time in multiple threads. Since, it's 
likely that iterating on the same object crash, I'm not sure that anyone do it.

If someone rely on the feature, we should maybe explain how to handle properly 
the issue: use a lock.

I have a preference for Serhiy's PR 3672. It's simple, the error is easy to 
understand. Maybe his PR just lacks a mention in "Porting to Python 2.7" 
section of What's New in Python 2.7 and/or the NEWS entry. Suggest to use a 
lock for example.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31530] Python 2.7 readahead feature of file objects is not thread safe

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 20.09.2017 17:22, Guido van Rossum wrote:
> 
>> Why not simply document the fact that read ahead in Python 2.7
>> is not thread-safe and leave it at that ?
> 
> Program bugs should not crash the interpreter. (ctypes excepted.)

Ideally not, agreed :-)

--
title: [2.7] Python 2.7 readahead feature of file objects is not thread safe -> 
Python 2.7 readahead feature of file objects is not thread safe

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com