Re: Problem caused by socket.setdefaulttimeout()

[Samuele Kaplun]

Hi Jay,

In data giovedì 8 agosto 2013 16:31:53, Luker, James ha scritto:
> >>> import socket
> >>> socket.setdefaulttimeout(40)
> >>> from multiprocessing import Manager
> >>> m = Manager()
> >>> res = m.list([])
> 
> Traceback ...

Uh! That's disturbing. that setdefaulttimeout call is there precisely to avoid 
endless sockets (the default on Python) which might lead to completely block 
the Python interpreter in case of synchronous operations going awry.

The fact that is set on a global level in bibdocfile module is due to this 
being the only possibility on Python 2.4 for the urllib module (the minimal 
Python version supported right now). 

Indeed in the next branch we might instead use a different approach since 
minimal Python version supported there is 2.6.

Could you add a ticket to describe this trouble so that we can start searching 
for a solution?

Cheers!
Sam
-- 
Samuele Kaplun
Invenio Developer ** 
INSPIRE Service Manager ** 

Re: Problem caused by socket.setdefaulttimeout()

[Lars Holm Nielsen]

Hey Jay,

Interesting problem. I can reproduce the problem on ZENODO production 
machines running Python 2.6.6 but I cannot reproduce on my Ubuntu 
installation with Python 2.7.4


Btw...you might want to look into Celery for distributed processing. 
Naturally I don't know if it would fit your use-case, but at least I've 
found celery much less headache than trying to use multiprocessing :-)


Best regards,
Lars

On 08.08.2013 22:31, Luker, James wrote:

Hi gang,

Does this stacktrace look familiar-ish to anyone?

Traceback (most recent call last):
  File "", line 1, in 
  File 
"/home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/managers.py", 
line 667, in temp

token, exp = self._create(typeid, *args, **kwds)
  File 
"/home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/managers.py", 
line 565, in _create

conn = self._Client(self._address, authkey=self._authkey)
  File 
"/home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/connection.py", 
line 175, in Client

answer_challenge(c, authkey)
  File 
"/home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/connection.py", 
line 413, in answer_challenge

message = connection.recv_bytes(256) # reject large message
IOError: [Errno 11] Resource temporarily unavailable

We started seeing exceptions like that yesterday when we tried 
incorporating a new blueprint into our ADS 2.0 Flask app that uses 
some components of the python mulitprocessing library. Googling turned 
up a few clues and mentions about socket timeout stuff. I suspected 
something we were importing was maybe monkey-patching or otherwise 
messing with sockets, so I painstakingly went through all of our 
import chains and tracked it back to bibdocfile.py*, where it sets 
"socket.setdefaulttimeout(40)". Apparently that's enough to totally 
break multiprocessing. I don't really understand why, but you can 
duplicate the error very easily...


>>> import socket
>>> socket.setdefaulttimeout(40)
>>> from multiprocessing import Manager
>>> m = Manager()
>>> res = m.list([])
Traceback ...

I was able to work around the exceptions by simply calling, 
socket.setdefaulttimeout(None), immediately after the "from invenio 
import ..." statements, but I figured I should give you a heads up. 
Also, Roman mentioned that you'd had problems making use of the 
multiprocessing library in the past. Surely it's not that simple though.


--jay

* our app imports some functions like run_sql from invenio



--
Lars Holm Nielsen
CERN, IT Department, Collaboration & Information Services
http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927



smime.p7s
Description: S/MIME Cryptographic Signature

Problem caused by socket.setdefaulttimeout()

[Luker, James]

Hi gang,

Does this stacktrace look familiar-ish to anyone?

Traceback (most recent call last):
  File "", line 1, in 
  File
"/home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/managers.py",
line 667, in temp
token, exp = self._create(typeid, *args, **kwds)
  File
"/home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/managers.py",
line 565, in _create
conn = self._Client(self._address, authkey=self._authkey)
  File
"/home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/connection.py",
line 175, in Client
answer_challenge(c, authkey)
  File
"/home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/connection.py",
line 413, in answer_challenge
message = connection.recv_bytes(256) # reject large message
IOError: [Errno 11] Resource temporarily unavailable

We started seeing exceptions like that yesterday when we tried
incorporating a new blueprint into our ADS 2.0 Flask app that uses some
components of the python mulitprocessing library. Googling turned up a few
clues and mentions about socket timeout stuff. I suspected something we
were importing was maybe monkey-patching or otherwise messing with sockets,
so I painstakingly went through all of our import chains and tracked it
back to bibdocfile.py*, where it sets "socket.setdefaulttimeout(40)".
Apparently that's enough to totally break multiprocessing. I don't really
understand why, but you can duplicate the error very easily...

>>> import socket
>>> socket.setdefaulttimeout(40)
>>> from multiprocessing import Manager
>>> m = Manager()
>>> res = m.list([])
Traceback ...

I was able to work around the exceptions by simply calling,
socket.setdefaulttimeout(None), immediately after the "from invenio import
..." statements, but I figured I should give you a heads up. Also, Roman
mentioned that you'd had problems making use of the multiprocessing library
in the past. Surely it's not that simple though.

--jay

* our app imports some functions like run_sql from invenio