Re: thread safe SMTP module
In article [EMAIL PROTECTED], Gordon Messmer [EMAIL PROTECTED] wrote: Aahz wrote: Assuming you have correctly tracked down the problem area, I would call that a thread bug in Python. But my experience is that you simply have run into a problem with the socket. I would suggest that using socket.setdefaulttimeout() would work just as well. I believe that solution, also would not work. This note is included in the socket documentation, regarding timeout mode: http://docs.python.org/lib/socket-objects.html A consequence of this is that file objects returned by the makefile() method must only be used when the socket is in blocking mode; in timeout or non-blocking mode file operations that cannot be completed immediately will fail. smtplib.SMTP uses file objects when reading SMTP responses. If I used setdefaulttimeout(), then the socket would be in timeout mode and the above note would be applicable. Hrm. At this point, I would suggest taking discussion to python-dev; it has been too long since I looked closely at thread/socket behavior. I am not at all above calling python's behavior a bug, except that it seemed like a known behavior given the note in the thread documentation regarding built-in functions that block on I/O. No, at this point I think this is neither bug nor about thread blocking on I/O. I think it's about sockets dying and the inability of sockets in blocking mode to recover. I have seen this kind of behavior in single-threaded systems. But it really needs someone who knows more than I do, and I think the first step here is to go ahead and file a bug report for tracking purposes. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ I disrespectfully agree. --SJM -- http://mail.python.org/mailman/listinfo/python-list
Re: thread safe SMTP module
Aahz wrote: That doesn't make any sense. Blocking I/O generally releases the GIL, which is the whole reason Python doesn't totally suck for threading. Nevertheless, among the caveats listed at http://docs.python.org/lib/module-thread.html is: Not all built-in functions that may block waiting for I/O allow other threads to run. (The most popular ones (time.sleep(), file.read(), select.select()) work as expected.) There may be other thread problems, but I doubt that you have correctly analyzed their source. I subclassed smtplib.SMTP and replaced only the lines of code that had to do with blocking IO (connect, send and receive operations). Beforehand, python would occasionally lock up. Having made those changes, python stopped locking up. I think the problem was pretty well apparent. I can't pin it down to which one of those three operations was at fault, and it may be that only one was. However, when I use non-blocking IO, the application works. When I used smtplib.SMTP, it didn't. I'm open to other explanations. -- http://mail.python.org/mailman/listinfo/python-list
Re: thread safe SMTP module
In article [EMAIL PROTECTED], Gordon Messmer [EMAIL PROTECTED] wrote: Aahz wrote: That doesn't make any sense. Blocking I/O generally releases the GIL, which is the whole reason Python doesn't totally suck for threading. Nevertheless, among the caveats listed at http://docs.python.org/lib/module-thread.html is: Not all built-in functions that may block waiting for I/O allow other threads to run. (The most popular ones (time.sleep(), file.read(), select.select()) work as expected.) That's why I said generally. There may be other thread problems, but I doubt that you have correctly analyzed their source. I subclassed smtplib.SMTP and replaced only the lines of code that had to do with blocking IO (connect, send and receive operations). Beforehand, python would occasionally lock up. Having made those changes, python stopped locking up. I think the problem was pretty well apparent. I can't pin it down to which one of those three operations was at fault, and it may be that only one was. However, when I use non-blocking IO, the application works. When I used smtplib.SMTP, it didn't. I'm open to other explanations. Assuming you have correctly tracked down the problem area, I would call that a thread bug in Python. But my experience is that you simply have run into a problem with the socket. I would suggest that using socket.setdefaulttimeout() would work just as well. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ I disrespectfully agree. --SJM -- http://mail.python.org/mailman/listinfo/python-list
Re: thread safe SMTP module
Aahz wrote: Assuming you have correctly tracked down the problem area, I would call that a thread bug in Python. But my experience is that you simply have run into a problem with the socket. I would suggest that using socket.setdefaulttimeout() would work just as well. I believe that solution, also would not work. This note is included in the socket documentation, regarding timeout mode: http://docs.python.org/lib/socket-objects.html A consequence of this is that file objects returned by the makefile() method must only be used when the socket is in blocking mode; in timeout or non-blocking mode file operations that cannot be completed immediately will fail. smtplib.SMTP uses file objects when reading SMTP responses. If I used setdefaulttimeout(), then the socket would be in timeout mode and the above note would be applicable. I am not at all above calling python's behavior a bug, except that it seemed like a known behavior given the note in the thread documentation regarding built-in functions that block on I/O. -- http://mail.python.org/mailman/listinfo/python-list
Re: thread safe SMTP module
In article [EMAIL PROTECTED], Gordon Messmer [EMAIL PROTECTED] wrote: I believe that I've seen this discussed previously, so maybe there's some interest in it. I wrote a threaded mail filtering framework a while ago, and one of the modules does address verification via SMTP. Since smtplib.SMTP uses blocking IO, it can block the whole interpreter. Sometimes the whole thing would stop working indefinitely. That doesn't make any sense. Blocking I/O generally releases the GIL, which is the whole reason Python doesn't totally suck for threading. There may be other thread problems, but I doubt that you have correctly analyzed their source. You can prove this for yourself by trying out my threaded spider at http://www.pythoncraft.com/OSCON2001/ThreadPoolSpider.py -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ I disrespectfully agree. --SJM -- http://mail.python.org/mailman/listinfo/python-list
thread safe SMTP module
I believe that I've seen this discussed previously, so maybe there's
some interest in it. I wrote a threaded mail filtering framework a
while ago, and one of the modules does address verification via SMTP.
Since smtplib.SMTP uses blocking IO, it can block the whole
interpreter. Sometimes the whole thing would stop working indefinitely.
I'm now aware that Twisted offers a non-blocking SMTP class, but I
didn't really want to make that a dependency of the address
verification. I ended up subclassing the smtplib.SMTP class and
rewriting the functions that do I/O. Perhaps someone who doesn't want
to install Twisted will find this class useful, someday. It doesn't
support TLS, but it is otherwise a thread-safe SMTP class.
class ThreadSMTP(smtplib.SMTP):
SMTP class safe for use in threaded applications.
This class reimplements the SMTP class with non-blocking IO,
so that threaded applications don't lock up.
This class won't make starttls support thread-safe.
def connect(self, host='localhost', port=0):
Connect to a host on a given port.
If the hostname ends with a colon (`:') followed by a number, and
there is no port specified, that suffix will be stripped off and the
number interpreted as the port number to use.
Note: This method is automatically invoked by __init__, if a host is
specified during instantiation.
if not port and (host.find(':') == host.rfind(':')):
i = host.rfind(':')
if i = 0:
host, port = host[:i], host[i+1:]
try: port = int(port)
except ValueError:
raise socket.error, nonnumeric port
if not port: port = smtplib.SMTP_PORT
if self.debuglevel 0: printsys.stderr, 'connect:', (host, port)
msg = getaddrinfo returns an empty list
self.sock = None
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
af, socktype, proto, canonname, sa = res
try:
self.sock = socket.socket(af, socktype, proto)
self.sock.setblocking(0)
if self.debuglevel 0: printsys.stderr, 'connect:', (host, port)
# Try to connect to the non-blocking socket. We expect connect()
# to throw an error, indicating that the connection is in progress.
# Use select to wait for the connection to complete, and then check
# for errors with getsockopt.
try:
self.sock.connect(sa)
except socket.error:
readySocks = select.select([self.sock], [], [], _smtpTimeout)
if self.sock in readySocks[0]:
soError = self.sock.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR)
if soError:
raise socket.error, 'connection failed, error: %d' % soError
else:
# The connection timed out.
raise socket.error, 'connection timed out'
except socket.error, msg:
if self.debuglevel 0: printsys.stderr, 'connect fail:', (host, port)
if self.sock:
self.sock.close()
self.sock = None
continue
break
if not self.sock:
raise socket.error, msg
(code, msg) = self.getreply()
if self.debuglevel 0: printsys.stderr, connect:, msg
return (code, msg)
def send(self, str):
Send `str' to the server.
if self.debuglevel 0: printsys.stderr, 'send:', repr(str)
if self.sock:
try:
# Loop: Wait for select() to indicate that the socket is ready
# for data, and call send(). If send returns a value smaller
# than the total length of str, save the remaining data, and
# continue to attempt to send it. If select() times out, raise
# an exception and let the handler close the connection.
while str:
readySocks = select.select([], [self.sock], [], _smtpTimeout)
if not readySocks[1]:
raise socket.error, 'Write timed out.'
sent = self.sock.send(str)
if sent len(str):
str = str[sent:]
else:
# All the data was written, break the loop.
break
except socket.error:
self.close()
raise smtplib.SMTPServerDisconnected('Server not connected')
else:
raise smtplib.SMTPServerDisconnected('please run connect() first')
def getreply(self):
Get a reply from the server.
Returns a tuple consisting of:
- server response code (e.g.
