Re: Confused about nested scopes and when names get added to namespaces
My tests were run in python 2.6.5. -- http://mail.python.org/mailman/listinfo/python-list
Confused about nested scopes and when names get added to namespaces
I'm having trouble understanding when variables are added to namespaces. I thought I understood it, but my nested function examples below have me very confused. In each test function below I have an x variable (so "x" is in the namespace of each test function). I also have a nested function in each (innerFunc) that has different flavors of trying to access or assign a variable named "x". --- def test1(): print "running test1..." x = 1 def innerFunc(): print "inner locals():", print "%s" % locals() # x not present (yet) x = 2 print x innerFunc() print "x left as %s\n" % x def test2(): print "running test2..." x = 1 def innerFunc(): print "inner locals():", print "%s" % locals() # x not present innerFunc() print "x left as %s\n" % x def test3(): print "running test3..." x = 1 def innerFunc(): print "inner locals():", print "%s" % locals() # how is x in locals in this case?? print x innerFunc() print "x left as %s\n" % x test1() test2() test3() --- With the nested scope rules, I thought that *at the time of* trying to access an object with a given name, if the name was not available in the namespace of the local function python would then check the namespace of the parent function. My tests above don't seem to match this. Specifically my "at the time of" assumption. What is happening in test3? How is it that "x" ends up in the local namespace before it is ever referenced? It seems that, prior to execution time, the python compiler is "magically" determining that I'm referencing a particular name before assignment and shoving it into the local namespace so it can be used. I did not think the compiler went into function bodies, aside from basic syntax checking. Further confusing me (or confirming the compiler behavior) is adding a 4th test with only one added line... def test4(): print "running test4..." x = 1 def innerFunc(): print "inner locals():", print "%s" % locals() # how is x in locals in this case?? print x x = 2 #ONLY ADDED LINE TO TEST3 innerFunc() print "x left as %s\n" % x In this case I get "UnboundLocalError: local variable 'x' referenced before assignment". I think this means that the compiler (prior to runtime) inspected the code, determined I will do an assignment, decided _not_ to bring the parent's x into the local namespace, and as a result caused the unbound name problem at runtime. It seems that the parent's "x" is brought directly into the local namespace (when appropriate), rather than the namespace lookup just moving up the hierarchy when not found. This is confusing to me and is making me question my understanding of namespace lookups. Are nested scopes a special case where the lookup is handled differently? What if I want to change the value of the parent's "x"? test4 implies that I can't. Have I got this right? Can someone please clarify what is going on? -- http://mail.python.org/mailman/listinfo/python-list
Re: * for generic unpacking and not just for arguments?
On Nov 29, 11:09 am, Christian Heimes wrote: > The feature is available in Python 3.x: > > >>> a, b, *c = 1, 2, 3, 4, 5 > >>> a, b, c > (1, 2, [3, 4, 5]) > >>> a, *b, c = 1, 2, 3, 4, 5 > >>> a, b, c > > (1, [2, 3, 4], 5) Interesting... especially the recognition of how both ends work with the "a, *b, c" example. That is some funky syntax. And it goes to a list instead of a tuple now, I see. It is also the opposite of what I was considering, although I expect it goes both ways like it does for functions? The given python 3.0 example is on the LHS with * packing many-to-one while unpacking (like *args inside a function), whereas I was referring to RHS-style unpacking where * explodes/unpacks one-to-many (like passing *args _to_ a function). I've steered clear of 3.x so far, but it looks like I'll have to give it a whirl if only to avoid asking irrelevant questions! -- http://mail.python.org/mailman/listinfo/python-list
* for generic unpacking and not just for arguments?
Is there a reason that this is fine: >>> def f(a,b,c): ... return a+b+c ... >>> f(1, *(2,3)) 6 but the code below is not? >>> x = (3, 4) >>> (1, 2, *x) == (1, 2, 3, 4) Traceback (most recent call last): File "", line 1, in invalid syntax: , line 1, pos 8 Why does it only work when unpacking arguments for a function? Is it because the code below is preferred, and more readable? >>> x = (3, 4) >>> (1, 2) + x == (1, 2, 3, 4) True I've rooted around to see if there is an answer already and found some threads going way back to 1998 (!!), but can't find a concise answer as to why it is limited to args. I don't have a burning desire for this to work, but I just tried it unsuccessfully when building up a tuple and was mildly surprised that it didn't work, so I'm curious why. Maybe it's just that * is strictly for arguments, and trying it for generic tuple unpacking is abuse (which is down the corridor in 12A). -- http://mail.python.org/mailman/listinfo/python-list
"proper"/best way to hack SimpleXmlRpcServer to support datetime?
I'm running python 2.5.1 and it seems that SimpleXmlRpcServer is not setup to support the base datetime module in the same way xmlrpclib has been with "use_datetime". I see that someone (Virgil Dupras) has recently submitted a fix to address this, but I don't want to patch my python distro. I want to work around it until the real upgrade comes. I have a klugey workaround I can just drop in my program that does the trick, but I'm curious if there is a better way? I have not done that many monkey patches. I'm also of course interested in someone telling me I don't need to patch this and I'm just blind to the correct way. My monkey patch is below... ### HACK HACK HACK ### ## Patching SimpleXmlRpcServer to support datetime (just like xmlrpclib does) from SimpleXMLRPCServer import xmlrpclib as _xmlrpclib _old_loads = _xmlrpclib.loads def _new_loads(*args, **kwargs): kwargs["use_datetime"] = True return _old_loads(*args, **kwargs) _xmlrpclib.loads = _new_loads ### ENDHACK ENDHACK ### That works, but are there any better, slicker, more pythonic, more robust, more interesting, or just plain more fun ways to do it? -- http://mail.python.org/mailman/listinfo/python-list
Re: Is crawling the stack "bad"? Why?
> OK, if you crawl the stack I will seek you out and hit you with a big > stick. Does that affect your decision-making? How big a stick? :) > Seriously, crawling the stack introduces the potential for disaster in > your program, since there is no guarantee that the calling code will > provide the same environment i future released. So at best you tie your > solution to a particular version of a particular implementation of Python. I'm gathering that the general argument is entirely centered around portability and future-proofing of code. This certainly makes sense. I could try and argue that that doesn't matter for write-once-change- never code, but anything I'd say there might as well be applied to an argument saying that writing crappy code is actually ok. And then I would need to be committed for thinking that write-once-change-never code actually exists. I'm making myself sick as I type this. > You might as well not bother passing arguments to functions at all, > since the functions could always crawl the stack for the arguments they > need.A major problem with this is that it constrains the caller to use > particular names for particular function arguments. You know, you're right! Arguments are overrated. All future code I write will be argument free. Just have to get rid of that pesky "self" now. I can't shake "'But I came here for an argument!' 'Oh... this is abuse'" from my mind. > What happens if two different functions need arguments of the same name? > Seriously, forget this craziness. I will (mostly)... I knew it was bad code and a total hack, I just was looking for a concise reason as to why. I appreciate the comments, guys... thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: Is crawling the stack "bad"? Why?
Thanks Ian... I didn't know about threading.local before but have been
experimenting and it will likely come in quite handy in the future.
For this particular case it does basically seem like a replacement for
the threadID indexed dictionary, though. ie: I'll still need to set
up the RpcContainer, custom request handler, and custom server in
order to get the info handed around properly. I will likely go with
this approach since it lets me customize other aspects at the same
time, but for client IP determination alone I still half think that
the stack crawler is cleaner.
No convincing argument yet on why crawling the stack is considered
bad? I kind of hoped to come out of this with a convincing argument
that would stick with me...
On Feb 25, 12:30 pm, Ian Clark <[EMAIL PROTECTED]> wrote:
> On 2008-02-25, Russell Warren <[EMAIL PROTECTED]> wrote:
>
>
>
> >> the threading.local class seems defined for that purpose, not that I've
> >> ever
> >> used it ;)
>
> > I hadn't heard of that... it seems very useful, but in this case I
> > think it just saves me the trouble of making a stash dictionary...
> > unless successive calls to threading.local return the same instance?
> > I'll have to try that, too.
>
> No, successive calls to threading.local() will return different objects.
> So, you call it once to get your 'data store' and then use that one
> object from all your threads. It takes care of making sure each thread
> gets it's own data.
>
> Here is your example, but using threading.local instead of your own
> version of it. :)
>
> Ian
>
> import xmlrpclib, threading, sys, thread
> from SimpleXMLRPCServer import SimpleXMLRPCServer,
> SimpleXMLRPCRequestHandler
>
> thread_data = threading.local()
>
> class RpcContainer(object):
> def __init__(self):
> self._Handlers = {} #keys = thread IDs, values=requestHandlers
> def _GetRpcClientIP(self):
> #connection = self._Handlers[thread.get_ident()].connection
> connection = thread_data.request.connection
> ip = connection.getpeername()[0]
> return ip
> def WhatIsMyIP(self):
> return "Your IP is: %s" % self._GetRpcClientIP()
>
> class ThreadCapableRequestHandler(SimpleXMLRPCRequestHandler):
> def do_POST(self, *args, **kwargs):
> #make the handler available to the RPCs, indexed by threadID...
> thread_data.request = self
> SimpleXMLRPCRequestHandler.do_POST(self, *args, **kwargs)
>
> class MyXMLRPCServer(SimpleXMLRPCServer):
> def __init__(self, RpcContainer, *args, **kwargs):
> self.RpcContainer = RpcContainer
> SimpleXMLRPCServer.__init__(self, *args, **kwargs)
>
> class DaemonicServerLaunchThread(threading.Thread):
> def __init__(self, RpcServer, **kwargs):
> threading.Thread.__init__(self, **kwargs)
> self.setDaemon(1)
> self.server = RpcServer
> def run(self):
> self.server.serve_forever()
>
> container = RpcContainer()
> rpcServer = MyXMLRPCServer(
> RpcContainer = container,
> addr = ("", 12390),
> requestHandler = ThreadCapableRequestHandler,
> logRequests = False)
> rpcServer.register_function(container.WhatIsMyIP)
> slt = DaemonicServerLaunchThread(rpcServer)
> slt.start()
>
> sp = xmlrpclib.ServerProxy("http://localhost:12390";)
> print sp.WhatIsMyIP()
--
http://mail.python.org/mailman/listinfo/python-list
Re: Is crawling the stack "bad"? Why?
> How about a dictionary indexed by by the thread name.
Ok... a functional implementation doing precisely that is at the
bottom of this (using thread.get_ident), but making it possible to
hand around this info cleanly seems a bit convoluted. Have I made it
more complicated than I need to? There must be a better way? It sure
is a heck of a lot less straightforward than having a reasonably tight
CrawlUpStackToGetClientIP function call. But then nothing is more
straightforward than a simple goto, either...
So I ask again, what is wrong with crawling the stack?
> What happens if you want to switch to pypy?
If it doesn't work if I decide to switch implementations for some
reason, I just fix it when my unit tests tell me it is busted. No?
Aren't there also python implementations that don't have threadign in
them that would file using thread.get_ident? Seems hard to satisfy
all implementations.
> the threading.local class seems defined for that purpose, not that I've ever
> used it ;)
I hadn't heard of that... it seems very useful, but in this case I
think it just saves me the trouble of making a stash dictionary...
unless successive calls to threading.local return the same instance?
I'll have to try that, too.
---
import xmlrpclib, threading, sys, thread
from SimpleXMLRPCServer import SimpleXMLRPCServer, \
SimpleXMLRPCRequestHandler
class RpcContainer(object):
def __init__(self):
self._Handlers = {} #keys = thread IDs, values=requestHandlers
def _GetRpcClientIP(self):
connection = self._Handlers[thread.get_ident()].connection
ip = connection.getpeername()[0]
return ip
def WhatIsMyIP(self):
return "Your IP is: %s" % self._GetRpcClientIP()
class ThreadCapableRequestHandler(SimpleXMLRPCRequestHandler):
def do_POST(self, *args, **kwargs):
#make the handler available to the RPCs, indexed by threadID...
self.server.RpcContainer._Handlers[thread.get_ident()] = self
SimpleXMLRPCRequestHandler.do_POST(self, *args, **kwargs)
class MyXMLRPCServer(SimpleXMLRPCServer):
def __init__(self, RpcContainer, *args, **kwargs):
self.RpcContainer = RpcContainer
SimpleXMLRPCServer.__init__(self, *args, **kwargs)
class DaemonicServerLaunchThread(threading.Thread):
def __init__(self, RpcServer, **kwargs):
threading.Thread.__init__(self, **kwargs)
self.setDaemon(1)
self.server = RpcServer
def run(self):
self.server.serve_forever()
container = RpcContainer()
rpcServer = MyXMLRPCServer( \
RpcContainer = container,
addr = ("", 12390),
requestHandler = ThreadCapableRequestHandler,
logRequests = False)
rpcServer.register_function(container.WhatIsMyIP)
slt = DaemonicServerLaunchThread(rpcServer)
slt.start()
sp = xmlrpclib.ServerProxy("http://localhost:12390";)
print sp.WhatIsMyIP()
--
http://mail.python.org/mailman/listinfo/python-list
Re: Is crawling the stack "bad"? Why?
> That is just madness. What specifically makes it madness? Is it because sys._frame is "for internal and specialized purposes only"? :) > The incoming ip address is available to the request handler, see the > SocketServer docs I know... that is exactly where I get the address, just in a mad way. > Write a request handler that stashes that info somewhere that rpc > responders can access it in a sane way. That is exactly where I started (creating my own request handler, snagging the IP address and stashing it), but I couldn't come up with a stash location that would work for a threaded server. This is the problem I was talking about with the "current_peer_info" scheme. How is the RPC responder function supposed to know what is the right stash, given that when threaded there could be multiple stashes at a time? The IP needs to get down to the exact function execution that is responding to the client... how do I do that? I had my options as: 1) stash the IP address somewhere where the RPC function could get it 2) pass the IP down the dispatch chain to be sure it gets to the target I couldn't come up with a way to get 1) to work. Then, trying to accomplish 2) I reluctantly started messing with different schemes involving my own versions of do_POST, _marshaled_dispatch, and _dispatch in order to pass the IP directly down the stack. After some pain at this (those dispatches are weird) I decided it was wy too much of a hack. Then I thought "why not go up the stack to fetch it rather than trying to mess with the nice/weird dispatch chain to send it down". I now had a third option... 3) Go up the stack to fetch the exact IP for the thread After realizing this I had my working stack crawl code only a few minutes later (I had GetCallerNameAndArgs already). Up the stack has a clear path. Down was murky and involved trampling on code I didn't want to override. The results is much cleaner than what I was doing and it worked, albeit with the as yet unfounded "crawling the stack is bad" fear still there. I should also point out that I'm not tied to SimpleXMLRPCServer, it is just a convenient example. I think any RPC protocol and dispatcher scheme would have the same problem. I'd be happy to hear about a clean stashing scheme (or any other alternative) that works for a threaded server. My biggest specific fear at the moment is that sys._frame will do funky things with multiple threads, but given that my toy example is executing in a server on its own thread and it traces perfectly I'm less worried. Come to think of it, I wonder what happens when you crawl up to and past thread creation? Hmm. -- http://mail.python.org/mailman/listinfo/python-list
Re: Is crawling the stack "bad"? Why?
Argh... the code wrapped... I thought I made it narrow enough. Here
is the same code (sorry), but now actually pasteable.
---
import SimpleXMLRPCServer, xmlrpclib, threading, sys
def GetCallerNameAndArgs(StackDepth = 1):
"""This function returns a tuple (a,b) where:
a = The name of the calling function
b = A dictionary with the arg values in order
"""
f = sys._getframe(StackDepth + 1) #+1 to account for this call
callerName = f.f_code.co_name
#get the arg count for the frame...
argCount = f.f_code.co_argcount
#get a tuple with the local vars in the frame (args first)...
localVars = f.f_code.co_varnames
#now get the tuple of just the args...
argNames = localVars[:argCount]
#now to make a dictionary of args and values...
argDict = {}
for key in argNames:
argDict[key] = f.f_locals[key]
return (callerName, argDict)
def GetRpcClientConnectionInfo():
#Move up the stack to the location to figure out client info...
requestHandler = GetCallerNameAndArgs(4)[1]["self"]
usedSocket = requestHandler.connection
return str(usedSocket.getpeername())
def StartSession():
return "RPC call came in on: %s" % GetRpcClientConnectionInfo()
class DaemonicServerLaunchThread(threading.Thread):
def __init__(self, RpcServer, **kwargs):
threading.Thread.__init__(self, **kwargs)
self.setDaemon(1)
self.server = RpcServer
def run(self):
self.server.serve_forever()
rpcServer = SimpleXMLRPCServer.SimpleXMLRPCServer(("", 12390), \
logRequests = False)
rpcServer.register_function(StartSession)
slt = DaemonicServerLaunchThread(rpcServer)
slt.start()
sp = xmlrpclib.ServerProxy("http://localhost:12390";)
print sp.StartSession()
--
http://mail.python.org/mailman/listinfo/python-list
Is crawling the stack "bad"? Why?
I've got a case where I would like to know exactly what IP address a
client made an RPC request from. This info needs to be known inside
the RPC function. I also want to make sure that the IP address
obtained is definitely the correct one for the client being served by
the immediate function call. That is kind of dumb/obvious to say, but
I do just to highlight that it could be a problem for an RPC server
allowing multiple simultaneous connections on multiple threads. ie: I
can't set some simple "current_peer_info" variable when the connection
is made and let the RPC function grab that value later since by the
time it does it could easily be wrong.
In order to solve this I toyed with a few schemes, but have (so far)
settled on crawling up the stack from within the RPC call to a point
where the precise connection info that triggered the RPC call to run
could be determined. This makes sure (I think!) that I get the exact
connection info in the event of a lot of simultaneous executions on
different threads. It seems hackish, though. I frequently find that
I take the long way around to do something only to find out later that
there is a nice and tight pythonic way to get it done. This seems
like it might be one of those cases and the back of my mind keeps
trying to relegate this into the realm of cheat code that will cause
me major pain later. I can't stop thinking the old days and slapping
gotos all over code to fix something "quickly" rather than
restructuring properly. Crawling around the stack in non-debugger
code always seems nasty to me, but it sure seems to work nicely in
this case...
To illustrate this scheme I've got a short program using
SimpleXMLRPCServer to do it. The code is below. If you run it you
should get an output something like:
RPC call came in on: ('127.0.0.1', 42264)
Does anyone have a better way of doing this? Anyone want to warn me
off of crawling the stack to get this type of info? The docstring for
sys._getframe already warns me off by saying "This function should be
used for internal and specialized purposes only", but without
providing any convincing argument why that is the case. I'd love to
hear a reasonable argument... the only thing I can think of is that it
starts dipping into lower level language behavior and might cause
problems if your aren't careful. Which is almost as vague as "for
internal and specialized purposes only".
I'm very curious to hear what you python wizards have to say.
import SimpleXMLRPCServer, xmlrpclib, threading, sys
def GetCallerNameAndArgs(StackDepth = 1):
"""This function returns a tuple (a,b) where:
a = The name of the calling function
b = A dictionary with the arg values in order
"""
f = sys._getframe(StackDepth + 1) #+1 to account for this call
callerName = f.f_code.co_name
#get the arg count for the frame...
argCount = f.f_code.co_argcount
#get a tuple with the local vars in the frame (puts the args
first)...
localVars = f.f_code.co_varnames
#now get the tuple of just the args...
argNames = localVars[:argCount]
#now to make a dictionary of args and values...
argDict = {}
for key in argNames:
argDict[key] = f.f_locals[key]
return (callerName, argDict)
def GetRpcClientConnectionInfo():
#Move up the stack to the right point to figure out client info...
requestHandler = GetCallerNameAndArgs(4)[1]["self"]
usedSocket = requestHandler.connection
return str(usedSocket.getpeername())
def StartSession():
return "RPC call came in on: %s" % GetRpcClientConnectionInfo()
class DaemonicServerLaunchThread(threading.Thread):
def __init__(self, RpcServer, **kwargs):
threading.Thread.__init__(self, **kwargs)
self.setDaemon(1)
self.server = RpcServer
def run(self):
self.server.serve_forever()
rpcServer = SimpleXMLRPCServer.SimpleXMLRPCServer(("", 12390), \
logRequests = False)
rpcServer.register_function(StartSession)
slt = DaemonicServerLaunchThread(rpcServer)
slt.start()
sp = xmlrpclib.ServerProxy("http://localhost:12390";)
print sp.StartSession()
--
http://mail.python.org/mailman/listinfo/python-list
Re: Looking for a good Python environment
> While we're at it, do any of these debuggers implement a good way to > debug multi-threaded Python programs? Wing now has multi-threaded debugging. I'm a big Wing (pro) fan. To be fair, when I undertook my huge IDE evaluation undertaking it was approx 2 years ago... at the time as far as what I would consider to be a full featured professional IDE it was IMO really only Wing and Komodo who could compete. The others were left in the dust. Unfortunately both cost money, but it became clear that at least in this instance you get what you pay for. Not a big deal for me because as far as professional development costs the cost is ridiculously low and I use it professionally, but I could see balking at the cost if strictly a hobbiest... although I would pay as I'd be lost without my Wing I think. At the time, I much preferred Wing to Komodo, but haven't tried Komodo more than sparingly since then. My bet is that the situation would still be similar since Wing has done nothing but get better over time. The support crew at Wing are great, too... the mailing list is excellent and the Wing developers typically respond very quickly to any support requests, and even feature requests (I've had a few things added due to the mailing list). The biggest missing feature in Wing at the moment is integrating GUI development. If you are into that, you may want to look elsewhere. Any GUI stuff I do I use wxPython and after starting with a template builder I just manually code the GUIs... painful at times, especially when you just want to whip up something small, but I've gotten used to it. Now that I type this, though, I think I'll go looking for what's new! Maybe Boa is less buggy now? Hmm. Prior to taking on my "find the ultimate IDE" quest I was using SPE and it was free and quite decent, just not comparable to Wing. http://pythonide.stani.be/ A quick look at the current state of SPE shows that it now has multi- threaded debugging via WinPDB (what I used to use for debugging thread issues). Interesting. Worth a look to see if it is integrated well. -- http://mail.python.org/mailman/listinfo/python-list
Re: logging module and trailing newlines
Both are very good responses... thanks! I had forgotten the ease of
"monkey-patching" in python and the Stream class is certainly cleaner
than the way I had been doing it.
On Oct 3, 3:15 am, Peter Otten <[EMAIL PROTECTED]> wrote:
> Russell Warren wrote:
> > All I'm after is the ability to log things like...
>
> > Compiling 'shrubbery.c'... [DONE]
>
> > where the "[DONE]" was added later in time than the "Compiling...", and
> > the output goes to both stdout and to a log file. ie: I want to tee my
> > print statements and keep the ability to skip the trailing newline. I had
> > rolled my own primitive version than decided to try the logging module for
> > kicks.
>
> > Anyone have a suggestion on how to get logging to work like this? Or know
> > of a way to tee in Windows without forcing other users to install a tee
> > package?
>
> (1) Logging
>
> If you are too lazy to subclass you can monkey-patch:
>
> >>> import logging
> >>> def emit(self, record):
>
> ... msg = self.format(record)
> ... fs = "%s" if getattr(record, "continued", False) else "%s\n"
> ... self.stream.write(fs % msg)
> ... self.flush()
> ...>>> logging.StreamHandler.emit = emit
> >>> continued = dict(continued=True)
> >>> logging.error("Compiling... ", extra=continued); logging.error("[Done]")
>
> ERROR:root:Compiling... ERROR:root:[Done]
>
> (2) Teeing
>
> "Primitive", but should work:
>
> >>> class Stream(object):
>
> ... def __init__(self, *streams):
> ... self.streams = streams
> ... def write(self, s):
> ... for stream in self.streams:
> ... stream.write(s)
> ... def flush(self):
> ... for stream in self.streams:
> ... stream.flush()
> ...>>> import sys
> >>> stream = Stream(sys.stdout, sys.stderr)
> >>> print >> stream, "Compiling...",
>
> Compiling...Compiling...>>>
>
> I'd probably go with the latter.
>
> Peter
--
http://mail.python.org/mailman/listinfo/python-list
logging module and trailing newlines
I was just setting up some logging in a make script and decided to give the built-in logging module a go, but I just found out that the base StreamHandler always puts a newline at the end of each log. There is a comment in the code that says "The record is then written to the stream with a trailing newline [N.B. this may be removed depending on feedback]"... I guess there wasn't the feedback to drive the change. All I'm after is the ability to log things like... Compiling 'shrubbery.c'... [DONE] where the "[DONE]" was added later in time than the "Compiling...", and the output goes to both stdout and to a log file. ie: I want to tee my print statements and keep the ability to skip the trailing newline. I had rolled my own primitive version than decided to try the logging module for kicks. Anyone have a suggestion on how to get logging to work like this? Or know of a way to tee in Windows without forcing other users to install a tee package? -- http://mail.python.org/mailman/listinfo/python-list
Re: Funky file contents when os.rename or os.remove are interrupted
Thanks, guys... this has all been very useful information. The machine this is happening on is already running NTFS. The good news is that we just discovered/remembered that there is a write-caching option (in device manager -> HDD -> properties -> Policies tab) available in XP. The note right beside the write-cache-enable checkbox says: "This setting enables write caching to improve disk performance, but a power outage or equipment failure might result in data loss or corruption." Well waddya know... write-caching was enabled on the machine. It is now disabled and we'll be power-cycle testing to see if it happens again. Regarding the comment on journaling file systems, I looked into it and it looks like NTFS actually does do journaling to some extent, and some effort was expended to make NTFS less susceptible to the exact problem I'm experiencing. I'm currently hopeful that the corrupted files we've seen are entirely due to the mistake of having write-caching enabled (the default). > Then, Windows has nothing to do with it, either. It calls the routines > of the file system driver rather directly. It looks like that is not entirely true... this write-caching appears to sit above the file system itself. In any case, it is certainly not a Python issue! One last non-python question... a few things I read seemed to vaguely indicate that the journaling feature of NTFS is an extension/option. Wording could also indicate a simple feature, though. Are there options you can set on your file system (aside from block size and partition)?! I've certainly never heard of that, but want to be sure. I definitely need this system to be as crash-proof as possible. Thanks again, Russ -- http://mail.python.org/mailman/listinfo/python-list
Funky file contents when os.rename or os.remove are interrupted
I've got a case where I'm seeing text files that are either all null characters, or are trailed with nulls due to interrupted file access resulting from an electrical power interruption on the WinXP pc. In tracking it down, it seems that what is being interrupted is either os.remove(), or os.rename(). Has anyone seen this behaviour, or have any clue what is going on? On first pass I would think that both of those calls are single step operations (removing/changing an entry in the FAT, or FAT-like thing, on the HDD) and wouldn't result in an intermediate, null-populated, step, but the evidence seems to indicate I'm wrong... Any insight from someone with knowledge of the internal operations of os.remove and/or os.rename would be greatly appreciated, although I expect the crux may be at the os level and not in python. Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: Intermittent "permission denied" errors when using os.rename and a recently deleted path??
> Does it actually tell you the target is the problem? I see an > "OSError: [Errno 17] File exists" for that case, not a permission error. > A permission error could occur, for example, if GDS has the source open > or locked when you call os.rename. No it doesn't tell me the target is the issue... you are of course right that it could be either. I did some looking to see if/why GDS would lock files at any time while scanning but didn't turn up anything useful so far. I'd be surprised if it did as that would be one heck of an annoying design flaw. Anyway - the retry-on-failure workaround seems to prevent it from happening, although it still seems very hackish and I don't like it: ... if os.path.exists(path1): os.remove(path1) startTime = time.clock() while 1: try: os.rename(self.path2, self.path1) break except OSError: if (time.clock() - startTime) > MAX_RETRY_DURATION_s: raise else: time.sleep(0) ... It feels very weird to have to verify a simple operation like this, but if it works it works. Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: Intermittent "permission denied" errors when using os.rename and a recently deleted path??
> Are you running a background file accessing tool like Google Desktop > Search or an anti-virus application? If so, try turning them off as a test. I'm actually running both... but I would think that once os.remove returns that the file is actually gone from the hdd. Why would either application be blocking access to a non-existent file? Of course, my thinking is obviously wrong since I do get the permission problem... I will definitely try disabling those. Now if only I could reproducably repeat it to make testing easier. :( Another thing is that I certainly do want the code to work in the presence of such tools. -- http://mail.python.org/mailman/listinfo/python-list
Re: Threads vs Processes
Oops - minor correction... xmlrpclib is fine (I think/hope). It is SimpleXMLRPCServer that currently has issues. It uses thread-unfriendly sys.exc_value and sys.exc_type... this is being corrected. -- http://mail.python.org/mailman/listinfo/python-list
Re: Threads vs Processes
> Another issue is the libraries you use. A lot of them aren't > thread safe. So you need to watch out. This is something I have a streak of paranoia about (after discovering that the current xmlrpclib has some thread safety issues). Is there a list maintained anywhere of the modules that are aren't thread safe? Russ -- http://mail.python.org/mailman/listinfo/python-list
Intermittent "permission denied" errors when using os.rename and a recently deleted path??
I've been having a hard time tracking down a very intermittent problem where I get a "permission denied" error when trying to rename a file to something that has just been deleted (on win32). The code snippet that gets repeatedly called is here: ... if os.path.exists(oldPath): os.remove(oldPath) os.rename(newPath, oldPath) ... And I get the permission denied exception on the os.rename line. Somehow the rename target is still locked? I don't get it. I found a post that seemed to refer to precisely this problem: http://groups.google.com/group/comp.lang.python/browse_frm/thread/496625ca3b0c3874/e5c19db11d8b6d4e?lnk=gst&q=os.remove+delay&rnum=1#e5c19db11d8b6d4e However - this post describes a case where there are multiple threads making use of other os calls. I am running a single threaded application and still getting this problem. ie: the suggested fix does not work for me. I'm trying to see if implementing a "trap the exception and try again, but not too many times" hack fix will do the trick, but I'm not a big fan of this "solution", and at this point I'm not entirely certain it will work because confirming that it *did* work is tough (it is very difficult to repeatably create the problem). Does anyone know of a real solution to this problem, or know what exactly is happening so that I can work out a proper solution? Thanks, Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: How to convert arbitrary objects directly to base64 without initial string conversion?
After some digging around it appears there is not a tonne of documentation on buffer objects, although they are clearly core and ancient... been sifting through some hits circa 1999, long before my python introduction. What I can find says that buffer is deprecated (Python in a Nutshell), or non-essential/for-older-versions (Python documentation). At least it no longer seems terribly weird to me that I never noticed this built-in before... I got this from the python docs in reference to buffer and others: "Python programmers, trainers, students and bookwriters should feel free to bypass these functions without concerns about missing something important". Is buffer safe to use? Is there an alternative? > ctypes objects support the buffer interface How can you tell what objects support the buffer interface? Is anything visible at the python level, or do you need to dig into the C source? Regarding documentation, I assume the C PyBufferObject is the underlying thing for the python-level buffer? If so, is the best place for docs on this ancient object to glean what I can from this link: http://www.python.org/doc/1.5.2p2/api/bufferObjects.html ? Any help is appreciated... I'd like to understand what I can about this object if I'm to use it... I'm wary of nasty surprises. Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: How to convert arbitrary objects directly to base64 without initial string conversion?
> Many functions that operate on strings also accept buffer objects as > parameters, > this seems also be the case for the base64.encodestring function. ctypes > objects > support the buffer interface. > > So, base64.b64encode(buffer(ctypes_instance)) should work efficiently. Thanks! I have never used (or even heard of) the buffer objects. I'll check it out. Russ -- http://mail.python.org/mailman/listinfo/python-list
How to convert arbitrary objects directly to base64 without initial string conversion?
I've got a case where I want to convert binary blocks of data (various ctypes objects) to base64 strings. The conversion calls in the base64 module expect strings as input, so right now I'm converting the binary blocks to strings first, then converting the resulting string to base64. This seems highly inefficient and I'd like to just go straight from binary to a base64 string. Here is the conversion we're using from object to string... import ctypes def ObjAsString(obj): sz = ctypes.sizeof(obj) charArray = ctypes.c_char * sz buf = charArray.from_address(ctypes.addressof(obj)) return buf.raw[:sz] The returned string can then be sent to base64 for conversion (although we're actually using xmlrpc.Binary), but there is obviously some waste in here. import base64 b64 = base64.b64encode(ObjAsString(foo)) Is there a canned/pre-existing way to convert a block of memory to a base64 string more efficiently? I'd like to avoid writing my own base64 conversion routine if possible. Anyone have any good ideas? Even a mroe efficient/less clunky way of conevrting an arbitrary object to a string would be appreciated. Thanks, Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Queue.Queue.queue.clear() thread-safe?
Thanks guys. This has helped decipher a bit of the Queue mechanics for me. Regarding my initial clear method hopes... to be safe, I've re-organized some things to make this a little easier for me. I will still need to clear out junk from the Queue, but I've switched it so that least I can stop the accumulation of new data in the Queue while I'm clearing it. ie: I can just loop on .get until it is empty without fear of a race, rather than needing a single atomic clear. My next Queue fun is to maybe provide the ability to stuff things back on the queue that were previously popped, although I'll probably try and avoid this, too (maybe with a secondary "oops" buffer). If curious why I want stuff like this, I've got a case where I'm collecting data that is being asynchronously streamed in from a piece of hardware. Queue is nice because I can just have a collector thread running and stuffing the Queue while other processing happens on a different thread. The incoming data *should* have start and stop indications within the stream to define segments in the stream, but stream/timing irregularities can sometimes either cause junk, or cause you to want to rewind the extraction a bit (eg: in mid stream-assembly you might realize that a stop condition was missed, but can deduce where it should have been). Fun. Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: better Python IDE? Mimics Maya's script editor?
Check out the Wing IDE - www.wingware.com . As part of it's general greatness it has a "debug probe" which lets you execute code snippets on active data in mid-debug execution. It doesn't have precisely what you are after... you can't (yet) highlight code segments and say "run this, please", but I think it might almost have what you want for general workflow improvement. The main drawback is that it is a commercial product, albeit "cheap". The extra drawback is that the debug probe feature requires the professional version which is "less cheap", but still < $200. Well worth it for professional development IMO. They have a great demo policy... you should check it out. I tried several different IDEs (I've become accustomed to using IDEs over supe'd up text editors) and Wing was/is my favorite. -- http://mail.python.org/mailman/listinfo/python-list
Is Queue.Queue.queue.clear() thread-safe?
I'm guessing no, since it skips down through any Lock semantics, but I'm wondering what the best way to clear a Queue is then. Esentially I want to do a "get all" and ignore what pops out, but I don't want to loop through a .get until empty because that could potentially end up racing another thread that is more-or-less blindly filling it asynchronously. Worst case I think I can just borrow the locking logic from Queue.get and clear the deque inside this logic, but would prefer to not have to write wrapper code that uses mechanisms inside the object that might change in the future. Also - I can of course come up with some surrounding architecture to avoid this concern altogether, but a thread-safe Queue clear would do the trick and be a nice and short path to solution. If QueueInstance.queue.clear() isn't thread safe... what would be the best way to do it? Also, if not, why is the queue deque not called _queue to warn us away from it? Any other comments appreciated! Russ -- http://mail.python.org/mailman/listinfo/python-list
Recommended way to fix core python distribution issues in your own apps?
I've got a case where I need to tweak the implementation of a default python library due to what I consider to be an issue in the library. What is the best way to do this and make an attempt to remain compatible with future releases? My specific problem is with the clock used in the threading.Event and threading.Timer. It currently uses time.time, which is affected by changes in system time. eg: if you change the system clock somehow at some time (say, with an NTP broadcast) you may get a surprise in the timing of your code execution. What I do right now is basically this: import sys import time import threading if sys.platform == 'win32': threading._time = time.clock in which case I'm simply forcing the internal clock used in the Event/Timer code to use a time-independent performance timer rather than the system time. I figured this is a much better way to do it than snagging a private copy of threading.py and making a direct change to it, but am curious if anyone has a better way of doing this type of thing? For example, I have no way of guaranteeing that this hack will work come a change to 2.5 or later. Thanks, Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: Popping from the middle of a deque + deque rotation speed
> So does the speed of the remaining 0.001 cases really matter? Note > that even just indexing into a deque takes O(index) time. It doesn't matter as much, of course, but I was looking to make every step as efficient as possible (while staying in python). As to indexing into a deque being O(index)... I didn't realize that. It is certainly something to keep in mind, though... looping through the contents of a deque would obviously be a bad idea with this being the case! I wonder if the generator for the deque helps reduce this? Will check later. Proof of the O(n) for indexing into a deque (not that I doubted Tim #2! :)... >>> import timeit >>> s = "from collections import deque; d = deque(xrange(100))" >>> timeit.Timer(stmt="x=d[1]", setup = s).timeit(number=10) 0.14770257113683627 >>> timeit.Timer(stmt="x=d[10]", setup = s).timeit(number=10) 1.4016418287799155 Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: Popping from the middle of a deque + deque rotation speed
Thanks for the responses. > It seems to work with my Python2.4 here. If you're > interested in efficiency, I'll leave their comparison as an > exercise to the reader... :) Ok, exercise complete! :) For the record, they are pretty much the same speed... >>> s = """ ... from collections import deque ... class mydeque(deque): ... def popmiddle(self, pos): ... self.rotate(-pos) ... ret=self.popleft() ... self.rotate(pos) ... return ret ... d = mydeque(xrange(100)) >>> timeit.Timer(stmt="x=d.popmiddle(1000)", setup = s).timeit(number=10) 5.4620059253340969 >>> s2=""" ... from collections import deque ... class mydeque(deque): ... def popmiddle(self, pos): ... ret = self[pos] ... del(self[pos]) ... return ret ... d = mydeque(xrange(100)) ... """ >>> timeit.Timer(stmt="x=d.popmiddle(1000)", setup = s2).timeit(number=10) 5.3937888754018104 Thanks for the alternative solution. Russ -- http://mail.python.org/mailman/listinfo/python-list
Popping from the middle of a deque + deque rotation speed
Does anyone have an easier/faster/better way of popping from the middle of a deque than this? class mydeque(deque): def popmiddle(self, pos): self.rotate(-pos) ret = self.popleft() self.rotate(pos) return ret I do recognize that this is not the intent of a deque, given the clearly non-"double-ended" nature. I'm using a deque in a place where 99.999 of the time it will be a fifo, but occasionally I will want to pop from the middle. I initially wrote that thinking that the rotate duration would be independent of the rotation distance, but... >>> import timeit >>> s = "from collections import deque; d = deque(xrange(100))" >>> timeit.Timer(stmt="d.rotate(1)", setup = s).timeit(number=10) 0.1372316872675583 >>> timeit.Timer(stmt="d.rotate(1000)", setup = s).timeit(number=10) 3.5050192133357996 >>> timeit.Timer(stmt="d.rotate(1)", setup = s).timeit(number=10) 32.756590851630563 >>> timeit.Timer(stmt="d.rotate(10)", setup = s).timeit(number=10) 325.59845064107299 >>> timeit.Timer(stmt="d.rotate(99)", setup = s).timeit(number=10) 0.14491059617921564 Boy was I wrong. Given that it scales linearly it looks like it cut-pastes the rotation an element at a time! At least it recognizes the shortest rotation path, though. On first guess of how the deque is implemented I would have thought that rotation could be achieved simply by diddling some pointers, but I could see how that would mess with popping efficiency (seems you'd have to remap memory in the event of a pop after rotation). Worst case I figured a rotate would just do a single shot memory remapping of the deque contents so that the speed was the same regardless of rotation size... My guessing/figuring skills clearly need some work. What's up with this deque rotation? If I were to hazard one more guess I'd think it is trying to conserve transient memory usage during rotation... in my (poor) mental scheme it seems that cutting/relocating could take 50% more memory than the deque itself for a full rotation. I should stop guessing. Or at least figure out how to find the source code for the deque implementation... Should I avoid using deques with large iterables? Thanks, Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: win32com short path name on 2k
I've been driven crazy by this type of thing in the past. In my case it was with the same application (not two like you), but on different machines, with all supposedly having the same OS load. In some cases I would get short path names and in others I would get long path names. I could never figure it out any logical explanation for the behaviour so I just worked around it (*cough* hack *cough*). To help you generally get around it, you may not know about these two functions (both in kernel32)... GetShortPathName - http://tinyurl.com/nxzkl GetLongPathName - http://tinyurl.com/r4ey4 You can work out a scheme with these where it doesn't matter what mood Windows is in when you ask it for a path. In my case I had another big problem, and that was that it would also arbitrarily decide to change the case of critical paths.. eg: "C:\Windows" on one machine, and "C:\windows" on another. That also drove me bonkers and resulted in some lost years/hair. -- http://mail.python.org/mailman/listinfo/python-list
Speed of shutil.copy vs os.system("copy src dest") in win32
I just did a comparison of the copying speed of shutil.copy against the speed of a direct windows copy using os.system. I copied a file that was 1083 KB. I'm very interested to see that the shutil.copy copyfileobj implementation of hacking through the file and writing a new one is significantly faster... any clue as to why this is? I figure I'm missing something here. Does os.system launch a cmd shell every time? >>> import timeit >>> timeit.Timer(stmt= r'shutil.copy(r"c:\windows\ntbtlog.txt", >>> r"c:\temp")',setup="import shutil").repeat(repeat=5,number=100) [0.99285104671434965, 0.68337121058721095, 0.84528340892575216, 0.87780765432398766, 0.8709894693311071] >>> timeit.Timer(stmt= r'os.system(r"copy c:\windows\ntbtlog.txt >>> c:\temp")',setup="import os").repeat(repeat=5,number=100) [2.8546278926514788, 2.3763950446300441, 2.609580241377, 2.4392499605455669, 2.4446956247265916] -- http://mail.python.org/mailman/listinfo/python-list
Re: Seems like I want a pre-processor, but...
Yes, I definitely should have done that for that case. I'm not entirely sure why I didn't. If I had, though, I may not have been prompted to ask the question and get all the other great little tidbits! -- http://mail.python.org/mailman/listinfo/python-list
Re: Seems like I want a pre-processor, but...
Thanks guys - all great responses that answered my question in a few different ways with the addition of some other useful tidbits! This is a nice summary: > In general the idea is to move the test from 'every time I need to do > something' to 'once when some name is defined'. Gotta love the response time of this newsgroup... Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: Seems like I want a pre-processor, but...
> the collections module was added in 2.4 Ah... sorry about that. I should have checked my example more closely. What I'm actually doing is rebinding some Queue.Queue behaviour in a "safe" location like this: def _get(self): ret = self.queue.popleft() DoSomethingSimple() return ret And self.queue.popleft() used to be self.queue.pop(0). self.queue is a deque object in 2.4 so I just used transferred that unluckily to the post in a poor attempt to simplify it - I had no clue that it was new. Should have made one up from scratch. Anyway - it worked... you've answered my question perfectly, thanks. I hadn't considered that the module loading phase could basically used for preprocessing. You even helped by subtly providing a better version checker... I didn't even know you could use < with a tuple like that. I'll have to look into the logic rules there. Russ -- http://mail.python.org/mailman/listinfo/python-list
Seems like I want a pre-processor, but...
After some digging it seems that python does not have any equivalent to C's #if directives, and I don't get it... For example, I've got a bit of python 2.3 code that uses collections.deque.pop(0) in order to pop the leftmost item. In python 2.4 this is no longer valid - there is no argument on pop (rightmost only now) and you call .popleft() instead. I would like my code to work in both versions for now and simply want to add code like: if sys.version[:3] == "2.3": return self.myDeque.pop(0) else: return self.myDeque.popleft() but am recoiling a bit at the unnecessary conditional in there that I think will be run on every execution - unless the compiler has some magic detection of things like sys.version to compile out the conditional as if it were a preprocessor directive (seems highly unlikely!)?. What is the pythonic thing to do? This is in a generally usable file, not a package, so I don't want to make a version for each (which seems to be what every package out there does). It seems to be begging for a pre-processor directive set. Thanks, Russ -- http://mail.python.org/mailman/listinfo/python-list
Profiling/performance monitoring in win32
The application we're working on at my company currently has about eleventy billion independent python applications/process running and talking to each other on a win32 platform. When problems crop up and we have to drill down to figure out who is to blame and how, we currently are using the (surprisingly useful) perfmon tool that comes with Windows. Perfmon does a pretty decent job, but is pretty raw and sparse on the usability front. Does anyone know of any alternative windows (not cpu) profilers out there? Commercial packages ok, Of course one that can introspect it's way into what a python app is doing would be a bonus (but truthfully I'm probably just adding that last bit to make this post a bit more appropriate for c.l.py :) ). Thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: Finding the public callables of self
> import inspect
> myCallables = [name for name, value in inspect.getmembers(self) if not
> name.startswith('_') and callable(value)]
Thanks. I forgot about the inspect module. Interestingly, you've also
answered my question more than I suspect you know! Check out the code
for inspect.getmembers():
def getmembers(object, predicate=None):
"""Return all members of an object as (name, value) pairs sorted by
name.
Optionally, only return members that satisfy a given predicate."""
results = []
for key in dir(object):
value = getattr(object, key)
if not predicate or predicate(value):
results.append((key, value))
results.sort()
return results
Seems familiar! The fact that this is using dir(), getattr(), and
callable() seems to tell me there is no better way to do it. I guess
my method wasn't as indirect as I thought!
And thanks for the reminder about getattr() instead of
__getattribute__() and other streamlining tips.
Russ
--
http://mail.python.org/mailman/listinfo/python-list
Finding the public callables of self
Is there any better way to get a list of the public callables of self
other than this?
myCallables = []
classDir = dir(self)
for s in classDir:
attr = self.__getattribute__(s)
if callable(attr) and (not s.startswith("_")):
myCallables.append(s) #collect the names (not funcs)
I don't mean a shorter list comprehension or something that just drops
the line count, but whether or not I need to go at it through dir and
__getattribute__. This seems a bit convoluted and with python it often
seems there's something already canned to do stuff like this when I do
it. At first I thought self.__dict__ would do it, but callable methods
seem to be excluded so I had to resort to dir, and deal with the
strings it gives me.
Thanks,
Russ
--
http://mail.python.org/mailman/listinfo/python-list
Re: Implied instance attribute creation when referencing a class attribute
Thanks for the additional examples, David (didn't see this before my last post). All of it makes sense now, including those examples. Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: Implied instance attribute creation when referencing a class attribute
D'oh... I just realized why this is happening. It is clear in the longhand as you say, but I don't think in the way you descibed it (or I'm so far gone right now I have lost it). self.I += 1 is the same as self.I = self.I + 1 and when python tries figures out what the 'self.I' is on the right hand side. it of course ends up having to move up to the base class foo.__dict__ because there is no 'I' in self.__dict__ yet. So it ends up effectively being: self.I = foo.I + 1 which explains where the "self.I = foo.I' that I was claiming was being done magically comes from. What my head was thinking was that the 'self.I' lookup would move up to get foo.__dict__['I'], and that I would effectively get 'foo.I += 1', but this is a bit of a brain fart and is just plain wrong. I should have seen that earlier... oh well. I'm happy that it is perfectly clear where it comes from, now. It still does look odd when you do a simplistic comparison of the behaviour of 'x += 1' and 'self.I += 1', but I suppose that that's just the way the lookup scheme crumbles. An unfortunate (and rare?) quirk, I guess. It still might be nice were python to just block out this potential confusion with an Exception... it seems that class vs instance attribute referencing is confusing enough for people without having this type of potential confusion lurking around the syntax. It seems like such a simple thing, but to understand the outcomes requires knowing how the name lookup scheme works, how mutable/immutable objects are dealt with, and what the += keystroke-saver/macro operator is actually doing. That this is stuff that someone coding in python should understand could certainly be argued, though... Russ -- http://mail.python.org/mailman/listinfo/python-list
Re: Implied instance attribute creation when referencing a class attribute
> I can see how this can be confusing, but I think the confusion here is > yours, not Pythons ;) This is very possible, but I don't think in the way you describe! > self.I += 10 is an *assignment*. Like any assignment, it causes the > attribute in question to be created ... no it isn't. The += is an operator. Look at the example I included with non_existent_var above. If that doesn't do it for you, pop open a clean python shell and do this one: >>> x += 2 Traceback (most recent call last): File "", line 1, in NameError: name 'x' is not defined Note that x doesn't exists and it does not create it. You can't normally operate on something before it is created - Python won't create it for you (which is why I was surprised by the class attribute behavior in the first post). > If you write out the longhand for += it becomes totally obvious what > is happening and why it makes sense: Not true as above. The longhand for 'self.I += 1' is 'self.I = self.I + 1', which normally needs self.I to exist due to the RHS of this. > So your case 1 is actually exactly what is happening! Python is > getting a hold of foo.I and incrementing it Nope. My case 1 would have the 'self.I += 1' modifying the actual class attribute, not some new instance attribute and this is definitely NOT happening. Maybe my example was bad? Try this one instead: >>> class foo(object): ... I = 1 ... def __init__(self): ... self.I += 123455 ... >>> a=foo() >>> a.I 123456 >>> foo.I 1 >>> del a.I >>> a.I 1 Note that we ended up binding a new "I" to the 'a' instance with the 'self.I += 1' statement, and it started with the value of 1 (the value of the base class attribute). I tried to make it clear in the example by wiping out the local copy, which then reveals the base class attribute when you go for it again. The fact that there is a local I being made with the value of the base class attribute says that Python is essentially adding the line 'self.I = foo.I' as in the code below. >>> class foo(object): ... I = 123455 ... def __init__(self): ... self.I = foo.I # unnecessary since python seems to do it in the next line ... self.I += 1 ... >>> a=foo() >>> b=foo() >>> c=foo() >>> print c.I, foo.I 123456 1 For kicks I added the b and c creations to show that at no time did the += operator get a hold of the foo base class as you state. It stayed untouched at 1 the whole time. To do that you need to reference foo itself as in the following case: >>> class foo(object): ... I = 0 ... def __init__(self): ... foo.I += 1 ... self.I = foo.I ... >>> a=foo() >>> b=foo() >>> c=foo() >>> print a.I, b.I, c.I, foo.I 1 2 3 3 >>> del a.I >>> a.I 3 Here it of course *did* increment the base foo attribute since it was directly referenced. 'a.I' stays as 1 here because I rebound a new instance attribute I on top with a copy of the base foo.I value due to it being immutable (a bit weird to use the same name, but I'm trying to show something) and it is what is retrieved first by Python (local dictionary first, if not found it goes to the base class). When I clear I from the local __dict__ with the del, you see that future self.I references skip out to the base class attribute since there is no instance I attribute anymore. A bit of a sidetrack there... still curious why python decides to auto-create the variable for you in this particular case. Any other takers? Russ -- http://mail.python.org/mailman/listinfo/python-list
Implied instance attribute creation when referencing a class attribute
I just ran across a case which seems like an odd exception to either
what I understand as the "normal" variable lookup scheme in an
instance/object heirarchy, or to the rules regarding variable usage
before creation. Check this out:
>>> class foo(object):
... I = 1
... def __init__(self):
... print self.__dict__
... self.I += 1
... print self.__dict__
...
>>> a=foo()
{}
{'I': 2}
>>> foo.I
1
>>> a.I
2
>>> del a.I
>>> a.I
1
>>> del a.I
Traceback (most recent call last):
File "", line 1, in
AttributeError: I
>>> non_existent_var += 1
Traceback (most recent call last):
File "", line 1, in
NameError: name 'non_existent_var' is not defined
In this case, 'self.I += 1' clearly has inserted a surprise
behind-the-scenes step of 'self.I = foo.I', and it is this which I find
interesting.
As I understand it, asking for self.I at this point should check
self.__dict__ for an 'I' entry, and if it doesn't find it, head on up
to foo.__dict__ and look for it.
So... I initially *thought* there were two possibilities for what would
happen with the 'self.I += 1':
1. 'self.I += 1' would get a hold of 'foo.I' and increment it
2. I'd get an AttributeError
Both were wrong. I thought maybe an AttributeError because trying to
modify 'self.I' at that point in the code is a bit fuzzy... ie: am I
really trying to deal with foo.I (in which case, I should properly use
foo.I) or am I trying to reference an instance attribute named I (in
which case I should really create it explicitly first or get an error
as with the non_existent_var example above... maybe with 'self.I =
foo.I').
Python is obviously assuming the latter and is "helping" you by
automatically doing the 'self.I = foo.I' for you. Now that I know this
I (hopefully) won't make this mistake again, but doing this seems
equivalent to taking my 'non_existent_var += 1' example above and
having the interpreter interpret as "oh, you must want to deal with an
integer, so I'll just create one for you with 'non_existent_var = 0'
first". Fortunately this is not done, so why do it with the instance
attribute reference?
Does anyone have any solid reasoning behind the Python behavior? It
might help drive it home more so than just taking it as "that's the way
it is" and remembering it.
It gets even more confusing for me because the behaviour could be
viewed as being opposite when dealing with mutable class members. eg:
>>> class foo(object):
... M = [1,2,3]
... def __init__(self):
... self.M.append(len(self.M) + 1)
... print self.M
...
>>> a=foo()
[1, 2, 3, 4]
>>> foo.M
[1, 2, 3, 4]
>>> del a.M
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'foo' object attribute 'M' is read-only
By opposite I mean that with immutable objects, a sloppy self.I
reference doesn't get you to the base class object, whereas with a
mutable one you do get to the base object (although I do recognize that
in both cases if you just remember that the interpreter will always
stuff in a 'self.x = BaseClass.x' it works as expected in both the
immutable and mutable case).
After all that, I guess it boils down to me thinking that the code
*should* interpret the attempted instance modification with one of the
two possibilities I mentioned above (although after typing this I'm now
leaning more towards an AttributeError rather than allowing 'self.I' to
be synonymous with 'foo.I' if no local override).
Russ
PS: Apologies if I mangled the "proper" terminology for talking about
this... hopefully it makes sense.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Threads and socket.setdefaulttimeout
Thanks for the detailed repsone... sorry for the lag in responding to it. After reading and further thought, the only reason I was using setdefaulttimeout in the first place (rather then using a direct settimeout on the socket) was because it seemed like the only way (and easy) of getting access to the seemingly deeply buried socket being used by xmlrpclib. That was prior to me using threads of course. I then started trying to make this solution work with thread, but it is now too convoluted as you say. Now I think the best solution is likely to redirect my efforts at getting access to the socket used by xmlrpclib so that I can set it's timeout directly. I'm still unclear how to do this cleanly, though. Getting to some of your comments. > When you say "one thread affects another", I see that your example uses > the same function for both threads. IMHO it's much better to override > the thread's run() method than to provide a callable at thread creating > time. That way you can be sure each thread's execution is firmly in the > context of the particular thread instance's namespace. > > having said all this, I don't think that's your issue. Correct - the bottom code is nothing to do with my code and was only to quickly prove that it was cross-thread. > This seems extremely contorted, and I'm pretty sure we can find a better > way. Couldn't agree more! > The threads' network calls should be yielding process control during > their timeout period to allow other runnable threads to proceed. That's Yep. This is not causing me any problem. > You are aware, I presume, that you can set a timeout on each socket > individually using its settimeout() method? Yes, but I momentarily had forgot about it... as mentioned I ended up making the since-bad choice of using setdefaulttimeout to get timeouts set on the inaccessible sockets. Then I carried it too far... > See above. However, this *does* require you to have access to the > sockets, which is tricky if they are buried deep in some opaque object's > methods. Any help on how to crack the safe would be appreciated. > There are locks! I suspect what you need is a threading.Rlock object, > that a thread has to hold to be able to modify the (global) default > timeout. This isn't a full solution to your problem, though, as you have > correctly deduced. Not quite what I was after I don't think since potentially interfering code needs to check the lock (via acquire) to avoid conflict. What I guess I mean is something general for the process saying "never ever interrupt this block og code by running code on another thread, regardless of whether the other thread(s) check a lock". Thinking more about it it seems unreasonable so I'll drop the question. Russ -- http://mail.python.org/mailman/listinfo/python-list
Threads and socket.setdefaulttimeout
It appears that the timeout setting is contained within a process
(thanks for the confirmation), but I've realized that the function
doesn't play friendly with threads. If I have multiple threads using
sockets and one (or more) is using timeouts, one thread affects the
other and you get unpredictable behavior sometimes. I included a short
script at the end of this that demonstrates the threading problem.
I'm trying to get around this by forcing all locations that want to set
a timeout to use a 'safe' call immediately prior to socket creation
that locks out setting the timeout again until the lock is released.
Something like this:
try:
SafeSetSocketTimeout(Timeout_s)
#lock currently acquired to prevent other threads sneaking in here
CreateSocket()
finally:
ReleaseSocketTimeoutSettingLock()
UseSocket()
However - this is getting increasingly painful because I don't have
easy access to all of the socket creations where I'd like to do this.
The biggest pain right now is that I'm using xmlrpclib which has some
seriously/frustratingly heavy use of __ prefixes that makes getting
inside to do this at socket creation near impossible (at least I think
so). Right now the best I can do is surround the xmlrpclib calls with
this (effectively putting the lock release after the UseSocket), but
then other threads get hung up for the duration of the call or timeout,
rather than just the simple socket creation.
It would be nice if the timeout were implemented as an argument in the
socket constructor rather than having this global method. Is there a
reason for this? I tried sifting through the cvs source and got lost -
couldn't even find the call definition for socket(family, type, proto)
and gave up...
Does anybody have any idea of another way to do what I need (indpendent
socket timeouts per thread), or have suggestions on how to break into
xmlrpclib (actually down into httplib) to do the methdo I was trying?
Related question: Is there some global way that I'm unaware of to make
it so that some few lines of code are atomic/uninterruptable and no
other thread can sneak in between?
All suggestions appreciated! Hopefully I'm just missing something
obvious.
Russ
#--- This script confirms that settimeout's affect is across threads
import threading, xmlrpclib, socket
def st():
socket.setdefaulttimeout(0.1)
try:
proxy = xmlrpclib.ServerProxy("http://localhost:1";)
print proxy.NonExistentCallThatShouldTimeout()
except Exception, E: print "Exception caught: %s" % (E,)
cbThread = threading.Thread(target = st)
cbThread.start()
try:
print proxy.NonExistentCallThatShouldTimeout()
except Exception, E: print "Exception caught: %s" % (E,)
#Output is:
#Exception caught: (10061, 'Connection refused')
#Exception caught: timed out
--
http://mail.python.org/mailman/listinfo/python-list
scope of socket.setdefaulttimeout?
Does anyone know the scope of the socket.setdefaulttimeout call? Is it a cross-process/system setting or does it stay local in the application in which it is called? I've been testing this and it seems to stay in the application scope, but the paranoid side of me thinks I may be missing something... any confirmation would be helpful. -- http://mail.python.org/mailman/listinfo/python-list
Re: Rollover/wraparound time of time.clock() under win32?
Thanks! That gets me exactly what I wanted. I don't think I would
have been able to locate that code myself.
Based on this code and some quick math it confirms that not only will
the rollover be a looong way out, but that there will not be any loss
in precision until ~ 30 years down the road. Checking my math:
(float(10**16 + 1) - float(10**16)) == 0
(float(10**15 + 1) - float(10**15)) == 1
ie: our double precision float can resolve unity differences out to
at least 10**15
Assuming 1 us/count we have 10**15 us / (3.15E13 us/year) = 31.7 yrs
Past this we won't roll over since the long keeps counting for a long
time, but some precision will be lost.
For those interested, the relevant win32 time code is below. Thanks
again!
time_clock(PyObject *self, PyObject *args)
{
static LARGE_INTEGER ctrStart;
static double divisor = 0.0;
LARGE_INTEGER now;
double diff;
if (!PyArg_ParseTuple(args, ":clock"))
return NULL;
if (divisor == 0.0) {
LARGE_INTEGER freq;
QueryPerformanceCounter(&ctrStart);
if (!QueryPerformanceFrequency(&freq) || freq.QuadPart == 0) {
/* Unlikely to happen - this works on all intel
machines at least! Revert to clock() */
return PyFloat_FromDouble(clock());
}
divisor = (double)freq.QuadPart;
}
QueryPerformanceCounter(&now);
diff = (double)(now.QuadPart - ctrStart.QuadPart);
return PyFloat_FromDouble(diff / divisor);
}
--
http://mail.python.org/mailman/listinfo/python-list
Rollover/wraparound time of time.clock() under win32?
Does anyone know how long it takes for time.clock() to roll over under win32? I'm aware that it uses QueryPerformanceCounter under win32... when I've used this in the past (other languages) it is a great high-res 64-bit performance counter that doesn't roll-over for many (many) years, but I'm worried about how many bits Python uses for it and when it will roll over. I need it to take years to roll over. I'm also aware that the actual rollover 'time' will be dependent on QueryPerformanceFrequency, so I guess my real question is how Python internally tracks the counter (eg: a 32 bit float would be no good), but the first question is easier to ask. :) time.time() is not an option for me since I need ms'ish precision and under win32 it is at best using the 18.2Hz interrupt, and at worst it is way worse. I'd like to avoid having to make direct win32api calls if at all possible. In general - if anyone has any other approaches (preferrably platform independent) for getting a < 1 ms resolution timer that doesn't roll over for years, I'd like to hear it! For now I have (for win32 and linux only): #--- from sys import platform if platform == 'win32': from time import clock as TimeStamp else: from time import time as TimeStamp print TimeStamp() #--- This works for me as long as the clock rollover is ok and precision is maintained. Thanks, Russ -- http://mail.python.org/mailman/listinfo/python-list
