[issue29167] Race condition in enum.py:_decompose()

[Ethan Furman]

Ethan Furman  added the comment:

The latest patch from issue38045 should make race-conditions non-existent.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Martin]

Martin  added the comment:

Our production system hit this issue using Python 3.6.7 once a few days ago, so 
presumably the race is still possible with the applied patch, just less likely?

```
RuntimeError: dictionary changed size during iteration
at _decompose (/usr/lib/python3.6/enum.py:858)
at _create_pseudo_member_ (/usr/lib/python3.6/enum.py:773)
at _missing_ (/usr/lib/python3.6/enum.py:764)
at __new__ (/usr/lib/python3.6/enum.py:535)
at __call__ (/usr/lib/python3.6/enum.py:293)
at __or__ (/usr/lib/python3.6/enum.py:800)
at create_urllib3_context 
(/usr/local/lib/python3.6/dist-packages/urllib3/util/ssl_.py:279)
at connect (/usr/local/lib/python3.6/dist-packages/urllib3/connection.py:332)
at _validate_conn 
(/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py:839)
at _make_request 
(/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py:343)
at urlopen 
(/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py:600)
at send (/usr/local/lib/python3.6/dist-packages/requests/adapters.py:449)
at send (/usr/local/lib/python3.6/dist-packages/requests/sessions.py:646)
at request (/usr/local/lib/python3.6/dist-packages/requests/sessions.py:533)
at __call__ 
(/usr/local/lib/python3.6/dist-packages/google/auth/transport/requests.py:120)
at _token_endpoint_request 
(/usr/local/lib/python3.6/dist-packages/google/oauth2/_client.py:106)
at jwt_grant 
(/usr/local/lib/python3.6/dist-packages/google/oauth2/_client.py:145)
at refresh 
(/usr/local/lib/python3.6/dist-packages/google/oauth2/service_account.py:322)
at before_request 
(/usr/local/lib/python3.6/dist-packages/google/auth/credentials.py:122)
at request 
(/usr/local/lib/python3.6/dist-packages/google/auth/transport/requests.py:198)
at wait_and_retry 
(/usr/local/lib/python3.6/dist-packages/google/resumable_media/_helpers.py:146)
at http_request 
(/usr/local/lib/python3.6/dist-packages/google/resumable_media/requests/_helpers.py:101)
at consume 
(/usr/local/lib/python3.6/dist-packages/google/resumable_media/requests/download.py:169)
at _do_download 
(/usr/local/lib/python3.6/dist-packages/google/cloud/storage/blob.py:498)
at download_to_file 
(/usr/local/lib/python3.6/dist-packages/google/cloud/storage/blob.py:560)
```

--
nosy: +gz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Roundup Robot]

Roundup Robot added the comment:

New changeset 95e184bd2d89 by Ethan Furman in branch '3.6':
closes issue29167: fix race condition in (Int)Flag
https://hg.python.org/cpython/rev/95e184bd2d89

New changeset e6b98c270718 by Ethan Furman in branch 'default':
issue29167: fix race condition in (Int)Flag
https://hg.python.org/cpython/rev/e6b98c270718

--
nosy: +python-dev
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Ethan Furman]

Ethan Furman added the comment:

Fixed the race condition for both the RuntimeError and for getting duplicate 
composite members.

--
keywords: +patch
stage: test needed -> patch review
Added file: http://bugs.python.org/file46394/issue29167.stoneleaf.01.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Ethan Furman]

Ethan Furman added the comment:

Thanks.  I'll go through and audit all my dictionary iterations.

--
nosy: +barry, eli.bendersky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Simon Percivall]

Simon Percivall added the comment:

Run this a couple of times (it fails for me the first time, but it's a race, so 
YMMV):

```
import enum
from concurrent.futures import ThreadPoolExecutor


class MyEnum(enum.IntFlag):
one = 1


with ThreadPoolExecutor() as executor:
print(list(executor.map(MyEnum.one.__or__, range(1000
```

An easy fix would be:

```
diff --git a/Lib/enum.py b/Lib/enum.py
index e79b0382eb..eca56ec3a7 100644
--- a/Lib/enum.py
+++ b/Lib/enum.py
@@ -846,7 +846,7 @@ def _decompose(flag, value):
 # check for named flags and powers-of-two flags
 flags_to_check = [
 (m, v)
-for v, m in flag._value2member_map_.items()
+for v, m in list(flag._value2member_map_.items())
 if m.name is not None or _power_of_two(v)
 ]
 members = []
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Ethan Furman]

Changes by Ethan Furman :


--
assignee:  -> ethan.furman
stage:  -> test needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Ethan Furman]

Ethan Furman added the comment:

Simon, can you post the exact line of code that causes the error?  It would be 
useful for creating a test case and the couple things I have tried to duplicate 
the error have worked fine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
components: +IO
nosy: +ethan.furman
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Simon Percivall]

New submission from Simon Percivall:

When called by `_create_pseudo_member_()`, the dictionary iteration of 
`_value2member_map` in `_decompose()` in enum.py may lead to a "RuntimeError: 
dictionary changed size during iteration". For me, it happened in `re.compile`.

```
Traceback (most recent call last):
  ...
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/re.py",
 line 233, in compile
return _compile(pattern, flags)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/re.py",
 line 301, in _compile
p = sre_compile.compile(pattern, flags)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_compile.py",
 line 562, in compile
p = sre_parse.parse(p, flags)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py",
 line 866, in parse
p.pattern.flags = fix_flags(str, p.pattern.flags)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/sre_parse.py",
 line 835, in fix_flags
flags |= SRE_FLAG_UNICODE
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 794, in __or__
result = self.__class__(self._value_ | self.__class__(other)._value_)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 291, in __call__
return cls.__new__(cls, value)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 533, in __new__
return cls._missing_(value)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 760, in _missing_
new_member = cls._create_pseudo_member_(value)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 769, in _create_pseudo_member_
_, extra_flags = _decompose(cls, value)
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 849, in _decompose
for v, m in flag._value2member_map_.items()
  File 
"/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py",
 line 848, in 
(m, v)
RuntimeError: dictionary changed size during iteration
```

--
messages: 284724
nosy: simon.percivall
priority: normal
severity: normal
status: open
title: Race condition in enum.py:_decompose()
type: crash
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com