[issue24555] Python logic error when deal with re and muti-threading

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> re functions never release GIL

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24555] Python logic error when deal with re and muti-threading

[R. David Murray]

R. David Murray added the comment:

If you re-post your bug information in a plain text and/or test program format 
it might get faster attention.

--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24555
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24555] Python logic error when deal with re and muti-threading

[bee13oy]

bee13oy added the comment:

#Python logic error when deal with re and muti-threading 
##Bug Description
  
When use re and multi-threading it will trigger the bug. 
  
Bug type:   `Logic Error`

Test Enviroment:

* `Windows 7 SP1 x64 + python 3.4.3`
* `Linux kali 3.14-kali1-amd64 + python 2.7.3 ` 

-Normal Case
- 1. main-thread: join(timeout), wait for sub-thread finished  -
- 2. sub-thread: while(1), an infinite loop-


Test Code:

#!/usr/bin/python
__author__ = 'bee13oy'
import re
import threading
timeout = 2
source = (.*(.)?)*bcd\\t\\n\\r\\f\\a\\e\\071\\x3b\\$\?caxyz
def run(source):
while(1):
print(test1)   
def handle():
try:
t = threading.Thread(target=run,args=(source,))
t.setDaemon(True)
t.start()
t.join(timeout)
print(thread finished...It's an normal case!\n)
except:
print(exception ...\n)
handle()

+

-Bug 
Case-
- 1. main-thread: join(timeout), wait for sub-thread finished   
 -
- 2. sub-thread: 1)we construct the special pattern 
(.*(.)?)*bcd\\t\\n\\r\\f\\a\\e\\071\\x3b\\$\?caxyz -
 2)regexp.search() can't deal with it, and hang 
up   -
 3)join(timeout), and the sub-thread was over 
time, at this time, main-thread should have got-
 the control of the program. But it didn't. 
 -
--

POC:

#!/usr/bin/python
__author__ = 'bee13oy'
import re
import os
import threading
timeout = 2
source = (.*(.)?)*bcd\\t\\n\\r\\f\\a\\e\\071\\x3b\\$\?caxyz
def run(source):
regexp = re.compile(r''+source+'')
sgroup = regexp.search(source)   
def handle():
try:
t = threading.Thread(target=run,args=(source,))
t.setDaemon(True)
t.start()
t.join(timeout)
print(finished...\n)
except:
print(exception ...\n)
handle()

+

-  Bug Analyze -

When we use Python multithreading, and use `join(timeout)` to wait until the 
**thread terminates** or **timed out**.   
1. In normal case, I run a while() in sub-thread, the main thread will 
get the control of the program after the sub-thread is timed out.
2. In our POC, even the sub-thread timed out, the main thread still 
can't execute continue. After analyzing, I found the main thread trapped into 
an infinite loop. 


At first, it will run into the sub-thread, but it can't end normally. 
At this time, join(timeout) will wait for the sub-thread return or timed out, 
and try to call timed out function in order that main thread can get the 
control of the program.

The bug is that the sub-thread was into an infinite loop and the main-thread 
was into an infinite loop too, which causes the program to be hang up.  

By analyzing the source code of Python, we found that:
  
- sub-thread is into an infinite loop   (code block 0)
- main-thread is into an infinite loop  (code block 1)
  
-code block 0-- 
- the following code is where sub-thread trapped into an infinite loop:  -
--- 
the following code is where the sub-thread trapped into an **infinite loop**:  
```
LOCAL(Py_ssize_t)
SRE(match)(SRE_STATE* state, SRE_CODE* pattern, int match_all)
{
SRE_CHAR* end = (SRE_CHAR *)state-end;
Py_ssize_t alloc_pos, ctx_pos = -1;
Py_ssize_t i, ret = 0;
Py_ssize_t jump;
unsigned int sigcount=0;
SRE(match_context)* ctx;
SRE(match_context)* nextctx;
TRACE((|%p|%p|ENTER\n, pattern, state-ptr));
DATA_ALLOC(SRE(match_context), ctx);
ctx-last_ctx_pos = -1;
ctx-jump = JUMP_NONE;
ctx-pattern = pattern;
ctx-match_all = match_all;
ctx_pos = alloc_pos;
.   
/* Cycle code which will never return*/
for (;;) {
++sigcount;
if ((0 == (sigcount  0xfff))  PyErr_CheckSignals())
RETURN_ERROR(SRE_ERROR_INTERRUPTED);

switch (*ctx-pattern++) {
case SRE_OP_MARK:
/* set mark */
/* MARK gid 

[issue24555] Python logic error when deal with re and muti-threading

[Matthew Barnett]

Matthew Barnett added the comment:

Your regex is a pathological case: it suffers from catastrophic backtracking 
and can take a long time to finish.

The other problem is that the re module never releases the GIL, so while it's 
performing the search in the low-level C code, other Python threads don't get a 
chance to run.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24555
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24555] Python logic error when deal with re and muti-threading

[bee13oy]

New submission from bee13oy:

Bug 0x01 is the main problem.

t.start()
t.join(timeout)
In normal case, I run a while() in sub-thread, the main thread will get the 
control of the program after the sub-thread is timed out.
But, in our POC, even the sub-thread timed out, the main thread still can't 
execute continue. After analyzing, I found the main thread trapped into an 
infinite loop like I described in the PDF.

--
components: Regular Expressions
files: python_logic_error.pdf
messages: 246138
nosy: bee13oy, ezio.melotti, mrabarnett
priority: normal
severity: normal
status: open
title: Python logic error when deal with re and muti-threading
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5
Added file: http://bugs.python.org/file39850/python_logic_error.pdf

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24555
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com