Re: MacOS 10.9.2: threading error using python.org 2.7.6 distribution
On Sunday, 27 April 2014 10:33:38 UTC-4, Chris Angelico wrote: > In most contexts, "thread unsafe" simply means that you can't use the > same facilities simultaneously from two threads (eg a lot of database > connection libraries are thread unsafe with regard to a single > connection, as they'll simply write to a pipe or socket and then read > a response from it). But processes and threads are, on many systems, > linked. Just the act of spinning off a new thread and then forking can > potentially cause problems. Those are the exact sorts of issues that > you'll see when you switch OSes, as it's the underlying thread/process > model that's significant. (Particularly of note is that Windows is > *very* different from Unix-based systems, in that subprocess > management is not done by forking. But not applicable here.) > Thanks, I'll keep all that in mind. I have to wonder how much of a problem it is here though, since I was able to demonstrate a functioning fork inside a new thread further up in the discussion. I have a new development that I find interesting, and I'm wondering if you still think it's the same problem. I have taken that threading object and turned it into a normal function definition. It's still forking the external tool, but it's doing so in the main thread, and it is finished execution before any other threads are created. And I'm still getting the same error. Turns out it's not coming from the threading module, but from the subprocess module instead. Specifically, like 709 of /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py which is this: try: self._execute_child(args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, to_close, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite) except Exception: I get the "Warning: No stack to get attribute from" twice when that self._execute_child() call is made. I've tried stepping into it to narrow it down further, but I'm getting weird behaviour from the debugger that I've never seen before once I do that. It's making it hard to track down exactly where the error is occurring. Interestingly, it's not actually raising an exception there. The except block is not being run. -- https://mail.python.org/mailman/listinfo/python-list
Re: MacOS 10.9.2: threading error using python.org 2.7.6 distribution
On Friday, 25 April 2014 14:58:56 UTC-4, Ned Deily wrote: > FWIW, the Python 2 version of subprocess is known to be thread-unsafe. > There is a Py2 backport available on PyPI of the improved Python 3 > subprocess module: Since that't the only thread that calls anything in subprocess, and I'm only running one instance of the thread, I'm not too concerned about how threadsafe subprocess is. In this case it shouldn't matter. Thanks for the info though.. that might be handy at some future point. -- https://mail.python.org/mailman/listinfo/python-list
Re: MacOS 10.9.2: threading error using python.org 2.7.6 distribution
On Friday, 25 April 2014 10:05:03 UTC-4, Chris Angelico wrote:
> First culprit I'd look at is the mixing of subprocess and threading.
> It's entirely possible that something goes messy when you fork from a
> thread.
I liked the theory, but I've run some tests and can't reproduce the error that
way. I'm using all the elements in my test code that the real code runs, and I
can't get the same error. Even when I deliberately break things I'm getting a
proper exception with stack trace.
class MyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
logger = logging.getLogger("thread")
p1 = subprocess.Popen( shlex.split( 'echo "MyThread calls echo."'),
stdout=subprocess.PIPE, universal_newlines=True)
logger.debug( p1.communicate()[0].decode('utf-8', 'ignore' ))
logger.debug( "MyThread runs and exits." )
def main():
console = logging.StreamHandler()
console.setFormatter(
logging.Formatter('%(asctime)s [%(name)-12s] %(message)s', '%T'))
logger = logging.getLogger()
logger.addHandler(console)
logger.setLevel(logging.NOTSET)
try:
t = MyThread()
#t = RTF2TXT("../data/SRD/rtf/", Queue.Queue())
t.start()
except Exception as e:
logger.error( "Failed with {!r}".format(e))
if __name__ == '__main__':
main()
> Separately: You're attempting a very messy charset decode there. You
> attempt to decode as UTF-8, errors ignored, and if that fails, you log
> an error... and continue on with the original bytes. You're risking
> shooting yourself in the foot there; I would recommend you have an
> explicit fall-back (maybe re-decode as Latin-1??), so the next code is
> guaranteed to be working with Unicode. Currently, it might get a
> unicode or a str.
Yeah, that was a logic error on my part that I hadn't got around to noticing,
since I'd been concentrating on the stuff that was actively breaking. That
should have been in an else: block on the end of the try.
--
https://mail.python.org/mailman/listinfo/python-list
MacOS 10.9.2: threading error using python.org 2.7.6 distribution
I've run into a threading error in some code when I run it on MacOS that works
flawlessly on a *BSD system running the same version of python. I'm running
the python 2.7.6 for MacOS distribution from python.org's downloads page.
I have tried to reproduce the error with a simple example, but so far haven't
been able to find the element or my code that triggers the error. I'm hoping
someone can suggest some things to try and/or look at. Googling for "pyton"
and the error returns exactly two pages, neither of which are any help.
When I run it through the debugger, I'm getting the following from inside
threading.start(). python fails to provide a stack trace when I step into
_start_new_thread(), which is a pointer to thread.start_new_thread(). It looks
like threading.__bootstrap_inner() may be throwing an exception which
thread.start_new_thread() is unable to handle, and for some reason the stack is
missing so I get no stack trace explaining the error.
It looks like thread.start_new_thread() is in the binary object, so I can't
actually step into it and find where the error is occurring.
> /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py(745)start()
-> _start_new_thread(self.__bootstrap, ())
(Pdb) s
> /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py(750)start()
-> self.__started.wait()
(Pdb) Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
Warning: No stack to get attribute from
My test code (which works) follows the exact same structure as the failing
code, making the same calls to the threading module's objects' methods:
import threading
class MyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
print "MyThread runs and exits."
def main():
try:
t = MyThread()
t.start()
except Exception as e:
print "Failed with {!r}".format(e)
if __name__ == '__main__':
main()
The actual thread object that's failing looks like this:
class RTF2TXT(threading.Thread):
"""
Takes a directory path and a Queue as arguments. The directory should be
a collection of RTF files, which will be read one-by-one, converted to
text, and each output line will be appended in order to the Queue.
"""
def __init__(self, path, queue):
threading.Thread.__init__(self)
self.path = path
self.queue = queue
def run(self):
logger = logging.getLogger('RTF2TXT')
if not os.path.isdir(self.path):
raise TypeError, "supplied path must be a directory"
for f in sorted(os.listdir(self.path)):
ff = os.path.join(self.path, f)
args = [ UNRTF_BIN, '-P', '.', '-t', 'unrtf.text', ff ]
logger.debug("Processing file {} with args {!r}".format(f, args))
p1 = subprocess.Popen( args, stdout=subprocess.PIPE,
universal_newlines=True)
output = p1.communicate()[0]
try:
output = output.decode('utf-8', 'ignore')
except Exception as err:
logger.error("Failed to decode output: {}".format(err))
logger.error("Output was: {!r}".format(output))
for line in output.split("\n"):
line = line.strip()
self.queue.put(line)
self.queue.put("")
Note: I only run one instance of this thread. The Queue object is used to pass
work off to another thread for later processing.
If I insert that object into the test code and run it instead of MyThread(), I
get the error. I can't see anything in there that should cause problems for
the threading module though... especially since this runs fine on another
system with the same version of python.
Any thoughts on what's going on here?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Two questions about logging
On Jan 12, 8:03 pm, K Richard Pixley wrote:
> Here's the confusion. Each log named __name__ is under the root logger.
> If you want them all, then catch them all with the root logger.
Thanks! I knew I was missing something obvious. Between you and Jean-
Michael Pichavant I've figured out what I need to do here.
On Jan 11, 9:34 pm, Roy Smith wrote:
> What I would do is log to syslog (logging.handlers.SysLogHandler) and
> let syslog worry about rotating log files. Why reinvent the wheel?
I've also worked out what I need to reset file handles, although it
took a lot of reading in the various logging.Handler subclasses. What
I needed isn't explicitly documented anywhere, but it turns out that
calling the close() method on a FileHandler instance does what I
need. There's no method to re-open the handler, but the next call to
emit() will automatically re-open the file if it isn't already open.
The upshot is that this does the expected thing if you rename its log
file and then send the running script a HUP signal.
#!/usr/bin/env python
import logging
import signal
import time
logger = logging.getLogger()
lh = logging.FileHandler('./foo.log')
lh.setFormatter(logging.Formatter('%(asctime)s %(name)s: %(message)s',
'%T'))
logger.addHandler(lh)
def sighup(signum, frame):
lh.close()
logger.error("handled {0}: {1}".format(signum, frame))
def main():
signal.signal(signal.SIGHUP, sighup)
while 1:
time.sleep(1)
logger.error('a message')
if __name__ == '__main__':
main()
--
http://mail.python.org/mailman/listinfo/python-list
Re: Two questions about logging
On Jan 11, 9:34 pm, Roy Smith wrote: > What I would do is log to syslog (logging.handlers.SysLogHandler) and > let syslog worry about rotating log files. Why reinvent the wheel? Syslog is fine for an application run by an administrator, but isn't an option for a user. -- http://mail.python.org/mailman/listinfo/python-list
Two questions about logging
I'm trying to figure out a couple of things with the logging module,
and I'm hoping someone can provide some pointers. I've read through
the module docs on python.org, the basic and advanced tutorials, and
the cookbook post, but a couple of things still elude me.
First, I'd like to be able to permit users to do more typical log
rotation, based on their OS's log rotation handler, rather than
rotating logs from inside an application. This is usually handled by
signalling an application with a HUP, whereupon it closes and then re-
opens all of its logs, getting new file handles (and new inodes). I
don't see anything in the Handler methods (or anywhere else) that
would let me tell a logger object to refresh the file handles on a log
file. Is there some standard way to deal with this?
Second, I'm trying to get a handle on how libraries are meant to
integrate with the applications that use them. The naming advice in
the advanced tutorial is to use __name__ to name loggers, and to allow
log messages to pass back up to the using application's logger for
processing, but these two pieces of advice seem contradictory.. since
log messages only pass back up to the root if the loggers are named
hierarchically.
> cat foo.py
#!/usr/bin/env python
import logging
import bar
logger = logging.getLogger(__name__)
fh = logging.FileHandler('foo.log')
ch = logging.StreamHandler()
logger.addHandler(fh)
logger.addHandler(ch)
logger.warning('from the first')
bar.funct()
> cat bar.py
#!/usr/bin/env python
import logging
def funct():
logger = logging.getLogger(__name__)
logger.warning('from the second')
> ./foo.py
from the first
No handlers could be found for logger "bar"
So, given this state of affairs, how is a library author to use
loggers, given that he or she can't know how authors who use the
library will name their logger objects? In the above example, what
would the author of bar.py do to hook up bar's logger with foo's,
without knowing in advance what foo's logger will be named?
Thanks very much for any suggestions, or pointers to documentation
that I've missed.
--
http://mail.python.org/mailman/listinfo/python-list
Re: cause __init__ to return a different class?
On Sep 15, 1:54 am, Ryan Kelly wrote: > To be friendlier to others reading your code, I would consider using a > classmethod to create an alternative constructor: I finally got back to looking at this today. As it turns out, un- overriding __new__ in the child class is more complicated than I first expected, and isn't worth the extra effort. So, I ended up using a constructor class method as you suggested. Thanks again! -- http://mail.python.org/mailman/listinfo/python-list
Re: cause __init__ to return a different class?
On Sep 15, 1:54 am, Ryan Kelly wrote: > The above will do exactly what you want, but it's generally bad style > unless you have a very specific use-case. Is there a particular reason > you need to "magically" return a subclass, rather than making this > explicit in the code? > > To be friendlier to others reading your code, I would consider using a > classmethod to create an alternative constructor: Yeah, I was considering doing this as well, particularly if I couldn't have made the other work. The reason I'm not too concerned about anyone misinterpreting what's going on is that in this case the base class is actually named for being a constructor, and any rare case where I want a specific subclass the subclass will be created directly. Thanks very much for your help! -- http://mail.python.org/mailman/listinfo/python-list
Re: cause __init__ to return a different class?
On Sep 15, 1:35 am, Chris Rebert wrote: > Override __new__() instead: > http://docs.python.org/reference/datamodel.html#object.__new__ Aha.. thanks! The reference book I'm working from neglects to mention __new__, so I'd assumed __init__ was the constructor. It hadn't occurred to me that python would separate the functions (I still don't see exactly why it would be useful to do that, but perhaps insight will come with time). -- http://mail.python.org/mailman/listinfo/python-list
cause __init__ to return a different class?
I'm wondering if there's a way in python to cause __init__ to return a class other than the one initially specified. My use case is that I'd like to have a superclass that's capable of generating an instance of a random subclass. I've tried both returning the subclass (as I would when overloading an operator) but I get the complaint that __init__ wants to return None instead of a type. The other thing I tried was overwriting 'self' while inside __init__ but that doesn't seem to work either. class Parent(object): def __init__(self, foo): if foo == True: self = Child(foo) class Child(Parent): def __init__(self, foo): pass Is there a way to do this? -- http://mail.python.org/mailman/listinfo/python-list
