[issue6824] help for a module should list supported platforms
Ritesh Raj Sarraf r...@researchut.com added the comment: I never said that I think it is not cross-platform. My point is, it should be good to list down a module's dependency in the python help docs. Like: tempfile Supported Platforms: ALL Exception: On Platform foo, feature tempfile.bar() is not available. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6824 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6824] help for a module should list supported platforms
Ritesh Raj Sarraf r...@researchut.com added the comment: Take help os or help os.fork for example. WIth the help output that they provide, how am I supposed to know that os.fork is only supported on Unix. We can also go with the assumption that the modules shipped are cross-platform. But then, for those that aren't, I don't think we mention that in the help file documentation. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6824 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6824] help for a module should list supported platforms
New submission from Ritesh Raj Sarraf r...@researchut.com: Whey I do a help (python-module), I get the help text as follows: Help on module tempfile: NAME tempfile - Temporary files. FILE /usr/lib/python2.5/tempfile.py MODULE DOCS http://www.python.org/doc/2.5.4/lib/module-tempfile.html DESCRIPTION This module provides generic, low- and high-level interfaces for creating temporary files and directories. The interfaces listed as safe just below can be used without fear of race conditions. Those listed as unsafe cannot, and are provided for backward compatibility only. The link to the webpage lists the supported platforms for the module. It would be good to have an entry stating the list of platforms supported by the module, in the module doc itself. -- assignee: georg.brandl components: Documentation messages: 92173 nosy: georg.brandl, rickysarraf severity: normal status: open title: help for a module should list supported platforms versions: Python 2.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6824 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6824] help for a module should list supported platforms
Changes by Ritesh Raj Sarraf r...@researchut.com: -- type: - feature request ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6824 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6247] should we include argparse
Ritesh Raj Sarraf r...@researchut.com added the comment: I'm not sure about the design part, but as a user, I find both to have very similar interfaces. argparse is better because it handles nargs=*. This has long been asked in optparse. Positional arguments is something I wanted recently, and argparse makes it very easy to use that too. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6247 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6247] should we include argparse
New submission from Ritesh Raj Sarraf r...@researchut.com: Shoudl argparse be included in the Python Standard Library. I know we already have getopt and optparse but optparse doesn't support many features easily. (like options without hyphen, nargs=*) Here a little about argparse: argparse: Python command line parser The argparse module makes writing command line tools in Python easy. Just briefly describe your command line interface and argparse will take care of the rest, including: parsing the arguments and flags from sys.argv converting arg strings into objects for your program formatting and printing any help messages and much more ... For those familiar with the optparse module from the Python standard library, argparse improves on this module in a number of ways, including: handling positional arguments supporting sub-commands allowing alternative option prefixes like + and / handling zero-or-more and one-or-more style arguments producing more informative usage messages providing a much simpler interface for custom types and actions -- components: Extension Modules messages: 89134 nosy: rickysarraf severity: normal status: open title: should we include argparse type: feature request versions: Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6247 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6247] should we include argparse
Ritesh Raj Sarraf r...@researchut.com added the comment: From the author, Steven Berthard: Sorry about the delay - conferences and moving means that I haven't had as much time for argparse as I'd like. Basically, the way things get into the Python standard library is if a large enough community requests it. I'm happy to contribute (and support) it if there's enough demand, though since Python 3.1b1 is already out, the earliest releases that might include it would be Python 2.7 and 3.2. From the conversation I have had with the author, he does seem to be interested in it. I'll ask the author to respond to this bug report. -- status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6247 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6247] should we include argparse
Ritesh Raj Sarraf r...@researchut.com added the comment: Thanks for the link. As a user, I see many of the features that argparse brings, to be helpful. Since they are missing in optparse, and since it doesn't look like argparse will be included, should I open new bugs for those features against optparse ? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6247 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Module imports during object instantiation
Steve Holden wrote: Ritesh Raj Sarraf wrote: On Aug 16, 12:16 am, Ritesh Raj Sarraf [EMAIL PROTECTED] wrote: On Aug 15, 11:42 pm, Neil Cerutti [EMAIL PROTECTED] wrote: [...] Oops!!! Looks like I completely missed this. It _did_ print the error message. Apologies to all for not keeping a close eye on the error message. Quite. It was obvious to many readers that you were fooling yourself with a false hypothesis. The probability of something so fundamental being wrong is vanishingly small. Yes, and now I realize how I would have got into the stupid assumption that imports were never executed in a class' __init__() was because: * I defined an import within a class' __init__() * It used to get executed. * Then I used to make an assumption that calling it from one of the methods would work. But instead the module was never in the Namespace. And that led me to the mis-understanding that the import never got executed. What a fool am I. I think people gave a good reason when they mentioned about top-level imports. I would move ahead with keeping things simple. Thanks to all for your time and patience on this thread (At least, no one is yelling at me ;-) Ritesh -- If possible, Please CC me when replying. I'm not subscribed to the list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Module imports during object instantiation
Neil Cerutti wrote: Doesn't __init__ get called automatically ? It gets called automatically when you construct an instance of the class in which it's defined. I am a little confused by your statements now. In my earlier posts in the same thread, I gave some code example which was something like this: class Log: def __init__(): if os.name == 'posix': try: import foobar except ImportError, e: print sys.stderr, e (Sorry for the bad indentation) Now, what do you mean here by construct an instance of the class in which it is defined ? __init__() _is_ defined in the Log class and I try to create an instance of the class Log as: log = Log() Assuming it is run on a box which is POSIX compliant, the try/import should fail and I should see an error message because there is no module named foobar. But I don't see that. And this is what exactly I wanted to get clarified in this whole thread. (But still haven't got a clear answer to a Yes/No). Or am I terribly missing something that you are trying to tell ? Ritesh -- If possible, Please CC me when replying. I'm not subscribed to the list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Module imports during object instantiation
On Aug 15, 11:42 pm, Neil Cerutti [EMAIL PROTECTED] wrote: On 2007-08-15, Ritesh Raj Sarraf [EMAIL PROTECTED] wrote: Or am I terribly missing something that you are trying to tell ? I didn't see log = Log() in your example. Sorry for the excursion. Are you sure os.name is 'posix' on your system? Here again for you: In [6]: cat rick.py IPython system call: cat rick.py import os, sys class Log: def __init__(self, verbose, lock = None): self.VERBOSE = bool(verbose) self.lock = bool(lock) if self.lock: self.dispLock = threading.Lock() else: self.dispLock = None if os.name == 'posix': try: import foobar except ImportError, e: print sys.stderr, e self.platform = 'posix' self.color = get_colors() elif os.name in ['nt', 'dos']: self.platform = 'microsoft' try: import SomeModule except ImportError, e: # comment out next line before going to prod... print sys.stderr, e self.color = None else: self.color = SomeModule.get_colors_windows() else: self.platform = None self.color = None In [7]: from rick import Log In [8]: log = Log(verbose = True) No module named foobar --- exceptions.NameError Traceback (most recent call last) /tmp/ipython console /tmp/rick.py in __init__(self, verbose, lock) 18 19 self.platform = 'posix' --- 20 self.color = get_colors() 21 22 elif os.name in ['nt', 'dos']: NameError: global name 'get_colors' is not defined In [9]: os.name Out[9]: 'posix' In [10]: import foobar --- exceptions.ImportError Traceback (most recent call last) /tmp/ipython console ImportError: No module named foobar In [11]: Okay!! Ignore line 20 for now because that is not what this whole thread has been about. In the code, just 2 lines above self.color = get_colors() you see the try/except import statements. Since there is no foobar named module, an ImportError should have been thrown and a relevant error message should have been printed. With these example I'm just trying to prove that imports put into a class' __init__() never get executed. They get executed if you put them in any of the methods. (And I don't know why it is done this way). Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: Module imports during object instantiation
On Aug 16, 12:16 am, Ritesh Raj Sarraf [EMAIL PROTECTED] wrote: On Aug 15, 11:42 pm, Neil Cerutti [EMAIL PROTECTED] wrote: On 2007-08-15, Ritesh Raj Sarraf [EMAIL PROTECTED] wrote: Or am I terribly missing something that you are trying to tell ? I didn't see log = Log() in your example. Sorry for the excursion. Are you sure os.name is 'posix' on your system? Here again for you: In [6]: cat rick.py IPython system call: cat rick.py import os, sys class Log: def __init__(self, verbose, lock = None): self.VERBOSE = bool(verbose) self.lock = bool(lock) if self.lock: self.dispLock = threading.Lock() else: self.dispLock = None if os.name == 'posix': try: import foobar except ImportError, e: print sys.stderr, e self.platform = 'posix' self.color = get_colors() elif os.name in ['nt', 'dos']: self.platform = 'microsoft' try: import SomeModule except ImportError, e: # comment out next line before going to prod... print sys.stderr, e self.color = None else: self.color = SomeModule.get_colors_windows() else: self.platform = None self.color = None In [7]: from rick import Log In [8]: log = Log(verbose = True) No module named foobar --- exceptions.NameError Traceback (most recent call last) /tmp/ipython console /tmp/rick.py in __init__(self, verbose, lock) 18 19 self.platform = 'posix' --- 20 self.color = get_colors() 21 22 elif os.name in ['nt', 'dos']: NameError: global name 'get_colors' is not defined Oops!!! Looks like I completely missed this. It _did_ print the error message. Apologies to all for not keeping a close eye on the error message. -- http://mail.python.org/mailman/listinfo/python-list
Re: Module imports during object instantiation
Bruno Desthuilliers wrote: What's leading you to conclude the import isn't being executed? You realise, I trust, that the module's code will only be executed on the first call to __init__()? Well. Putting it in a try inside __init__() doesn't do anything. This would be highly suprising. Yes, it is surprising. Unfortunately. :-( Or maybe I'm doing something really dumb. The import never happens. As soon as your running this code on a platform where os.name yields either 'nt' or 'dos', the import statement is executed. You can bet your ass on this. Now this import can fail for a lot of reasons (imported module not in sys.path, error in the imported module, etc), and since your catching *and dismissing* the ImportError, you don't know what effectively happens. It's like someone send you mails, you set your filter to trash mails from this person, and then complain this person never send you mails !-) And thus if I make a call in any of the methods, it fails with an error message. A NameError IIRC. IIRC ? Inspecting the traceback may help, you know. There's no traceback because the import never happens. Here's what I've captured for you exactly I see on my terminal [EMAIL PROTECTED]:/tmp$ cat rick.py class Log: def __init__(self, verbose, lock = None): self.VERBOSE = bool(verbose) self.lock = bool(lock) if self.lock: self.dispLock = threading.Lock() else: self.dispLock = None if os.name == 'posix': try: import foobar except ImportError, e: print sys.stderr, e self.platform = 'posix' self.color = get_colors() elif os.name in ['nt', 'dos']: self.platform = 'microsoft' try: import SomeModule except ImportError, e: # comment out next line before going to prod... print sys.stderr, e self.color = None else: self.color = SomeModule.get_colors_windows() else: self.platform = None self.color = None [EMAIL PROTECTED]:/tmp$ python Python 2.4.4 (#2, Jul 21 2007, 11:00:24) [GCC 4.1.3 20070718 (prerelease) (Debian 4.1.2-14)] on linux2 Type help, copyright, credits or license for more information. import os, sys os.path.abspath(os.curdir) '/tmp' from rick import Log import rick os.name 'posix' import foobar Traceback (most recent call last): File stdin, line 1, in ? ImportError: No module named foobar For the 'posix' OS, I'm doing an erratic import which should fail and print the error message. See, it prints nothing. This _is_ what has made me conclude that imports aren't executed in __init__(). But hey, they will get executed if you put them in any of the methods (which is again illogical because methods can/are called multiple times). Ritesh -- If possible, Please CC me when replying. I'm not subscribed to the list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Module imports during object instantiation
Neil Cerutti wrote: If you want an import inside an __init__ to run, you must call the __init__ function that contains it. Doesn't __init__ get called automatically ? I thought __init__ was required to be called explicitly only when you were doing inheritance and wanted to pass separate values to the base class. Ritesh -- If possible, Please CC me when replying. I'm not subscribed to the list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Module imports during object instantiation
Bruno Desthuilliers wrote: Ritesh Raj Sarraf a écrit : if lock is None or lock != 1: self.DispLock = False else: self.DispLock = threading.Lock() self.lock = True if os.name == 'posix': self.platform = 'posix' self.color = get_colors() elif os.name in ['nt', 'dos']: self.platform = 'microsoft' try: import SomeModule except ImportError: self.Set_Flag = None if self.Set_Flag is not None: self.color = SomeModule.get_colors_windows() else: self.platform = None self.color = None When I create an object the import part never gets executed. Is there a reason behind it ? what does print os.name yields ? On Windows: nt On Linux: posix I mean I'd like to keep my class as independent as I want. So that when later I need to use it somewhere else, I don't need to know if it depends on any modules. Then pass the module to the initializer. Python's modules are objects... Yes, that's an option. But is that the only one? To me it looks like an ugly way of doing. Ritesh -- If possible, Please CC me when replying. I'm not subscribed to the list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Module imports during object instantiation
Bruno Desthuilliers wrote: Ritesh Raj Sarraf a écrit : The initializer will be called *each time* you instanciate the class. And nothing prevents client code from calling it explicitelly as many times as it wants - ok, this would be rather strange, but this is still technically possible. What I mean that you have no assurance about the number of times an initializer will be called. Yes, it should be called _each time_ I do an instantiation. But the point is, I'm doing it only once. And I don't see people instantiating multiple times. And even if you do, it should do separate imports. log1 = Log() log2 = Log() Both the objects are separate from each other in every manner and they share nothing. wrt/ your problem, remember that top-level code is executed when the module is loaded (either as a main program or as an imported module). The canonical solution to os-specific imports is to handle them at the top-level: if os.name == 'posix: from some_posix_module import get_colors elif os.name in ['nt', 'dos']: from some_nt_module import get_windows_color as get_colors else: get_colors = lambda: None # or any other sensible default value... class SomeClass(object): def __init__(self, *args, **kw): self.colors = get_colors() This is what I'm left with to do currently. But I doubt if that makes by classes independent and to Just Work. If someone was to cut/paste just the class, it won't work. He'll have to go through the imports and figure out which one relates to the class he want's to import and add similar code to his. Ritesh -- If possible, Please CC me when replying. I'm not subscribed to the list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Module imports during object instantiation
Steve Holden wrote: Ritesh Raj Sarraf wrote: class Log: def __init__(self, verbose, lock = None): if verbose is True: self.VERBOSE = True else: self.VERBOSE = False Better: self.VERBOSE = verbose or, if you suspect verbose might pass in a mutable value, self.VERBOSE bool(verbose) Thanks for pointing this out. What's leading you to conclude the import isn't being executed? You realise, I trust, that the module's code will only be executed on the first call to __init__()? Well. Putting it in a try inside __init__() doesn't do anything. The import never happens. And thus if I make a call in any of the methods, it fails with an error message. A NameError IIRC. You are right in assuming that __init__() is called once per instance created, and it's legitimate to make an import conditional in the way you have because of the execute code only once behavior - if the module is already in sys.modules then it won't be re-imported, the existing one will be used. This is what even my understanding is. But am afraid to say that this understanding is not implemented in Python. Have you tried doing this? Does it work for you? Having said all that, I still don't see why you can't just put the try/except at the top level of your code and have color be a global. Why repeat the attempted import and the computation for each object you create? Alternatively, do it at the class level, so it's only executed once when the class is declared? Hmmm!! This is where I might not have done my homework. I just tried to import the same class (Log) the following way and it worked. from module import Log It worked. Now I have some questions. Going with your point of try/except imports at the top level, I am having multiple imports at the top level for the multiple classes that I have in the module. Not everything from the top level imports is required for the class Log (Say, just one module is what is required). So when I do a `from module import Log`, do all the modules at the top level get imported? My understanding says Yes, and if that's true, that's bad IMO. (Sorry! I'd really not thought about this when reading about Python module imports) My ultimate goal is to make all my classes as independent as possible while keeping them as lightweight as possible. If all top level module import statements are getting executed during a `from module import class`, this is _not_ lightweight IMO. That's what led me to try imports in __init__() but unfortunately I haven't seen that working for me even once. Ritesh -- If possible, Please CC me when replying. I'm not subscribed to the list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Module imports during object instantiation
On Aug 11, 3:17 am, James Stroud [EMAIL PROTECTED] wrote: You do realize your import statement will only be called for nt and dos systems don't you? Yes. I would like to load a Windows Python Module (which is, say a specific implementation for Windows only) in such a condition where I find that the platform is Dos/NT. Loading it globally doesn't make sense because when from the same class an object is created on a non-windows platfom, it would be waste. It would either ImportError or else just load the module and never use it. As per my understanding __init__ is executed only once during object creation. For such situations (that I've explained above), I think __init__() is the correct place to put imports. Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Module imports during object instantiation
Hi, I've been very confused about why this doesn't work. I mean I don't see any reason why this has been made not to work. class Log: def __init__(self, verbose, lock = None): if verbose is True: self.VERBOSE = True else: self.VERBOSE = False if lock is None or lock != 1: self.DispLock = False else: self.DispLock = threading.Lock() self.lock = True if os.name == 'posix': self.platform = 'posix' self.color = get_colors() elif os.name in ['nt', 'dos']: self.platform = 'microsoft' try: import SomeModule except ImportError: self.Set_Flag = None if self.Set_Flag is not None: self.color = SomeModule.get_colors_windows() else: self.platform = None self.color = None When I create an object the import part never gets executed. Is there a reason behind it ? I mean I'd like to keep my class as independent as I want. So that when later I need to use it somewhere else, I don't need to know if it depends on any modules. Currently, the way I'm left is to globally go and import the module and set a flag there. Ritesh -- If possible, Please CC me when replying. I'm not subscribed to the list. -- http://mail.python.org/mailman/listinfo/python-list
Namespace issue
Hi, I need a little help in understanding how Namespaces and scoping works with Classes/Functions in Python. Here's my code: class FetchData: def __init__(self, dataTypes=[foo, bar, spam], archive=False): self.List = [] self.Types = dataTypes if archive: self.Archiver = Archiver(True) def FetchData(self, PackageName, Filename=None): try: import my_module except ImportError: return False if Filename != None: try: file_handle = open(Filename, 'a') except IOError: sys.exit(1) (amnt, header, self.List) = my_module.get_data(PackageName) This is the only way this code will work. As per my understanding, the bad part is that on every call of the method FetchData(), an import would be done. To not let that happen, I can put the import into __init__(). But when I put in there, I get a NameError saying that my_module is not available even though it got imported. All I noticed is that the import has to be part of the method else I end up getting a NameError. But always importing my_module is also not good. What is the correct way of doing this ? IMO, ideally it should be part of __init__() and be imported only when the class is instantiated. Thanks, Ritesh -- If possible, Please CC me when replying. I'm not subscribed to the list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Debugging multithreaded program using Eclipse/Pydev
John Henry wrote: From what I can gather, it appears the only real option I have is to debug under Eclipse/Pydev. I did a google search of this newsgroup and didn't turn up too many hits. Before I invest the time to learn Eclipse/Pydev, I like to hear from somebody that have gone this path. Have you been successful in using Eclipse/Pydev to debug multi- threaded Python applications? Is so, what was the learning curve like to you? I use pydev extensively for all my Python related work. And I really love it. The good part about Eclipse/PyDev is that (if you are an Eclipse user) you'll have to learn only a single IDE for all your programming works in most of the languages. The interface, the shortcut keys to step, all are the same across all languages that you use in Eclipse. As for PyDev, it works very good with Eclipse. Be it code-completion or thread debugging, all is supported. For multithreaded applications, in PyDev, you'll see all the threads listed. Then you can use each thread and proceed with debugging the code. Learning Eclipse might take some time but that's worth it. HTH, Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: lock problem
Leo Kislov wrote: But you miss the fact that there is only one environment per process. Oh!! I think I get your point. There'd be only one __kabc_ldap environment variable to which all the threads would be overwriting. Hmmm!! Yes, you're correct. Thanks for pointing it out. Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: lock problem
Leo Kislov wrote: But you miss the fact that there is only one environment per process. Maybe there's a confusion. The environment variable that I'm setting has noting to do with ldapsearch. I use the environment variable as a filename to which ldapsearch can redirect its output. And that I do is because the output can be huge and useless. Then I do some pattern matching on that file and filter my data and then delete it. If you think I still am missing something important, request you to describe it. Thanks, Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: lock problem
Gabriel Genellina wrote: En Thu, 15 Mar 2007 18:31:29 -0300, Ritesh Raj Sarraf [EMAIL PROTECTED] escribió: I'm not sure if there's something wrong in the code mentioned above or is it really a lock problem. Try to break the code into smaller pieces to see what is wrong. It's too long for somebody to try to understand it. The only break I can do is to remove threading, and the it works fine. I also noticed that when debugging the threads (when the program stops at a breakpoint in the thread), data is written properly. This is what made me think if this really is a locking problem. Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: lock problem
Leo Kislov wrote: You're changing environmental variable __kabc_ldap that is shared between your threads. Environment is not designed for that kind of usage, it was designed for settings. Either use an option to set output file or just redirect stdout. If the interface of ldapsearch is so lame that it requires environmental variable use env to set the variable: env __kabc_ldap=/tmp/wrjhdsf ldapsearch ... The environment variable is set with temp_file_name which gets the name from tempfile.mkstemp(), which is run in every thread. So I don't think the environment variable is going to be the same. Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
lock problem
Hi,
I think there is some lock problem.
Let me show the code first
import os
import sys
import string
import tempfile
import threading
import Queue
from time import sleep
_hostname = None
_username = None
_password = None
_ldapDataFile = None
if _hostname is None or _username is None or _password is None or _ldapDataFile
is None:
sys.stderr.write(Please first set the credentials properly.\n)
sys.exit(1)
_ldapPreCommand = 'ldapsearch -LLL -z 0 (Sn='
_ldapPostCommand = '*) -h ' + _hostname + ' -x -D ' + _username + ' -b
Cn=users,DC=hq,DC=domain,DC=com' + -w + _password
NumberOfThreads = 5
#char = 's'
#ldapCommand = _ldapPreCommand + char + _ldapPostCommand
#x = os.system(ldapCommand)
#if x == 1024:
#print I'm in the exception
#print x
#sys.exit(1)
ldap_attributes =
['dn:', 'cn:', 'sn:', 'l:', 'st:', 'title:', 'description:', 'postalCode:',
'telephoneNumber:', 'facsimileTelephoneNumber:',
'givenName:', 'mail:', 'homePhone:', 'mobile:', 'pager:']
try:
writeFile = open(_ldapDataFile, 'w')
except IOError:
sys.stderr.write(Couldn't open file %s to write.\n % (writeFile) )
sys.exit(1)
#print Writing data to %s\n % (temp_file_name)
def RecordFetcher(char):
(temp_file_fd, temp_file_name) = tempfile.mkstemp()
os.environ['__kabc_ldap'] = temp_file_name
sleep(5) #Let's not thrash the exchange server ;-)
ldapCommand = _ldapPreCommand + char + _ldapPostCommand
if os.system(ldapCommand + $__kabc_ldap) != 0:
sys.stderr.write(Couldn't execute the command %s\n %
(ldapCommand) )
sys.exit(1)
#temp_file_name = /tmp/tmpnhYrps
try:
readFile = open(temp_file_name, 'r')
except IOError:
sys.stderr.write(Couldn't open file %s to read.\n % (readFile) )
sys.exit(1)
for record in readFile.readlines():
if record.startswith(' '): # Remove the junk
pass
record = string.rstrip(record, \n)
for attrib in ldap_attributes:
if record.startswith(attrib):
try:
FileLock.acquire(True)
if ldap_attributes[0] == attrib: #This attribute is
common/mandatory in all records, so we can rely on it
writeFile.write(\n)
writeFile.write(record)
writeFile.write(\n)
finally:
writeFile.flush()
FileLock.release()
break
readFile.close()
os.remove(temp_file_name)
#writeFile.write(\n)
def run(request, response, func=RecordFetcher):
while 1:
item = request.get()
if item is None:
break
(char, subChar) = item
response.put(func(char+subChar) )
# Start from here
requestQueue = Queue.Queue()
responseQueue = Queue.Queue()
FileLock = threading.Lock()
thread_pool = [
threading.Thread(
target=run,
args=(requestQueue, responseQueue)
)
for i in range(NumberOfThreads)
]
for t in thread_pool: t.start()
for char in string.lowercase:
# I know this is ugly. Too many cycles
# But ldapsearch or exchange is restricting, the query max result limit is
1000
for subChar in string.lowercase:
requestQueue.put( (char, subChar) )
for t in thread_pool: requestQueue.put(None)
for t in thread_pool: t.join()
writeFile.close()
=
Now as per the above code, aa is the first string which will be executed in
Thread-1. In my query to the ldap server, I am getting a record which matches
the aa string. I've verified it by putting a breakpoint and checking the
value.
The problem is that when I run the program manually, I don't get the data from
the first thread i.e. of the string aa.
I'm not sure if there's something wrong in the code mentioned above or is it
really a lock problem.
Can somebody please help about where I'm doing any mistake ?
Thanks,
Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
Necessity is the mother of invention.
Stealing logic from one person is plagiarism, stealing from many is research.
The great are those who achieve the impossible, the petty are those who
cannot - rrs
--
http://mail.python.org/mailman/listinfo/python-list
Re: screen output problem
I'm not sure if there is a definite solution to this problem. I've noticed that one of the applications, which I use on a daily basis (apt from Debian) does address the progress bar issue in another way. When apt tries to download multiple files, it displays the progress of all the downloads on a single line. Probably the apt developers also might have run into the same issue and hence settled down with this workaround. Thanks, Ritesh Dennis Lee Bieber wrote: On Tue, 28 Nov 2006 15:15:28 +0530, Ritesh Raj Sarraf [EMAIL PROTECTED] declaimed the following in comp.lang.python: If there's a way to print multiple lines of text withouth the newline (\n) character, we can then use carriage return (\r) and implement a proper progressbar with a bar for each separate progress action. If you were running on an Amiga, or via a serial port connection to an old VT-100 terminal, it would be child's play... Since both understood the same terminal control codes for moving the cursor around a text display. The old MS-DOS ANSI console driver may have understood that set. But M$, in its great and all-powerful wisdom, seems to feel that the only need for a command line interface is for batch configuration scripts that run with no human interaction, and hence with no need for status. One is expected to code a GUI instead if such things as progress-bars are needed. A google for windows console cursor control brings up (a bit of a surprise since I my search terms didn't mention Python): http://newcenturycomputers.net/projects/wconio.html Of course, this isn't portable to LINUX or other operating systems... Also, looking at win32api, I find a SetCursorPos() (and matching get...). -- WulfraedDennis Lee Bieber KD6MOG [EMAIL PROTECTED] [EMAIL PROTECTED] HTTP://wlfraed.home.netcom.com/ (Bestiaria Support Staff: [EMAIL PROTECTED]) HTTP://www.bestiaria.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: screen output problem
On Sunday 26 November 2006 13:07, Calvin Spealman wrote: Take a look at this: http://cheeseshop.python.org/pypi/progressbar Hi, Sorry for being a little late in replying. The progressbar implementation is excellent but it has the same problems that the current progressbar implementation I use, has. In the above mentioned progressbar implementation also, the progressbar status is overwritten if you're using multiple threads. If your application is threaded, and multiple threads are downloading multiple files, you get a progressbar which is overwritten by all the 3 threads. I had brought this issue some time back too. Dennis Lee Bieber had a solution which looked better but not very much. His implementation had os.system(clear/cls) being called (to update the progress) which made the screen flicker if your network latency was low (especially in cases where the data is indirectly being downloaded from a Proxy or a product like NetApp's Netcache). If there's a way to print multiple lines of text withouth the newline (\n) character, we can then use carriage return (\r) and implement a proper progressbar with a bar for each separate progress action. Thanks, Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs pgp4GnEOe0frx.pgp Description: PGP signature -- http://mail.python.org/mailman/listinfo/python-list
screen output problem
Hi, I have, for very long, been trying to find a consistent solution (which could work across major python platforms - Linux, Windows, Mac OS X) for the following problem. I have a function which downloads files from the web. I've made the function threaded. I'm trying to implement a progress bar for it which could work across all the platforms. The problem is that when the progress bar is displayed, all the threads overwrite the progress bar. I've thought of using curses but AFAIK curses is not available for Windows. Is there a generic way to accomplish the following: progress = [===]\n[===] for x in range(5): sys.stdout.write(progress + \r) Basically, I want a way through which I want to display data on two lines and make sure that \r overwrites the two lines without putting a newline making sure that whatever library it uses is available across all major Python platforms. Thanks, Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: thread lock question
I'm sorry. My example code wasn't clear enough. Please see the following: exit_status = copy_first_match(foo, bar) if exit_status == False: if download_file(foo, bar) == True: if zip_bool: ziplock.acquire() try: compress_the_file(zip_type_file, foo, bar) os.unlink(foo) # Unlink it once it has been zipped finally: ziplock.release() else: if zip_bool: ziplock.acquire() try: compress_the_file(zip_type_file, foo, bar) os.unlink(foo) finally: ziplock.release() I think this code should be clear enough. The program, before trying to download from the web, checks into its cache repository to see if the file is available or not. If available, it copies it from the local cache repository and adds to the archive, else it downloads from the web and archives it. Now in this scenario, with threads, say a file of 100 mb got downloaded and then being zipped. It might take a couple of seconds for the zip to complete. During those couple of seconds, if another thread's file (a couple kb) gets downloaded/copied, will it wait for the lock to be released or will it create another lock ? Thanks, Ritesh Dennis Lee Bieber wrote: On Wed, 11 Oct 2006 18:39:32 +0530, Ritesh Raj Sarraf [EMAIL PROTECTED] declaimed the following in comp.lang.python: if download_file(foo, bar) == True: if some condition: do_something() if zip_bool: snip if zip_bool: snip Basically, I'm testing for some condition, if that is True, do something and then do the zipping. I'm also doing the zipping even if that's not true. Actually, given the code you show, if some condition is true, you are /attempting/ to zip the file twice; the second one likely fails due to the first one deleting the file -- a condition masked by the finally: clause. All you really need there is if download_file(foo, bar): if some condition: do_something() if zip_bool: My question is, in a code example like this which is threaded, does the locking mechanism work correctly ? Or are two different locks being acquired ? Is the lock object itself defined globally (shared by the threads)? -- WulfraedDennis Lee Bieber KD6MOG [EMAIL PROTECTED] [EMAIL PROTECTED] HTTP://wlfraed.home.netcom.com/ (Bestiaria Support Staff: [EMAIL PROTECTED]) HTTP://www.bestiaria.com/ -- http://mail.python.org/mailman/listinfo/python-list
thread lock question
Hi, I have a question regarding locks in threads. Following is the code example: if download_file(foo, bar) == True: if some condition: do_something() if zip_bool: ziplock.acquire() try: compress_the_file(zip_type_file, foo, bar) os.unlink(foo) # Unlink it once it has been zipped finally: ziplock.release() if zip_bool: ziplock.acquire() try: compress_the_file(zip_type_file, foo, bar) os.unlink(foo) finally: ziplock.release() Basically, I'm testing for some condition, if that is True, do something and then do the zipping. I'm also doing the zipping even if that's not true. My question is, in a code example like this which is threaded, does the locking mechanism work correctly ? Or are two different locks being acquired ? Thanks, Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Threads and Progress Bar
Hi, I have a small application, written in Python, that uses threads. The application uses function foo() to download files from the web. As it reads data from the web server, it runs a progress bar by calling an install of a progress bar class. When using threads, I get the problem that the progress bar gets over-written by the download progress of files from other threads. I believe my change has to go into the progress bar class to make it thread aware. Are they any docs/suggestions on how to implement progress bars along with threads ? Thanks, Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: Threads and Progress Bar
): '''Get items from the request Queue, process them with func(), put the results along with the Thread's name into the response Queue. Stop running once an item is None.''' while 1: item = request.get() if item is None: break (sUrl, sFile, download_size, checksum) = stripper(item) response.put((name, sUrl, sFile, func(sUrl, sFile, sSourceDir, None))) # This will take care of making sure that if downloaded, they are zipped (thread_name, Url, File, exit_status) = responseQueue.get() if exit_status == True: if zip_bool: ziplock.acquire() try: compress_the_file(zip_type_file, File, sSourceDir) os.remove(File) # Remove it because we don't need the file once it is zipped. finally: ziplock.release() else: variables.errlist.append(File) pass # Create two Queues for the requests and responses requestQueue = Queue.Queue() responseQueue = Queue.Queue() # Pool of NUMTHREADS Threads that run run(). thread_pool = [ threading.Thread( target=run, args=(requestQueue, responseQueue) ) for i in range(NUMTHREADS) ] # Start the threads. for t in thread_pool: t.start() # Queue up the requests. for item in lRawData: requestQueue.put(item) # Shut down the threads after all requests end. # (Put one None sentinel for each thread.) for t in thread_pool: requestQueue.put(None) # Don't end the program prematurely. # # (Note that because Queue.get() is blocking by # defualt this isn't strictly necessary. But if # you were, say, handling responses in another # thread, you'd want something like this in your # main thread.) for t in thread_pool: t.join() Second... It sounds like you only created one progress bar, and each thread is referencing that single bar. I'd suspect you need to create a bar for EACH thread you create, and tell the thread which bar to update. Yes, you're correct. That's what I'm also suspecting. I tried to do some minor changes but couldn't succeed. Request you to, if you reply with code, give a little explanation so that I can understand and learn from it. Thanks, Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Bryan Olson on Saturday 05 Aug 2006 13:31 wrote: Exactly. Only one thread can hold a lock at a time. In the code above, a form called a critical section, we might think of a thread as holding the lock when it is between the acquire() and release(). But that's not really how Python's locks work. A lock, even in the locked state, is not held by any particular thread. If a thread tries to acquire a lock that some other thread has, it'll wait until the other thread releases it. More accurate: If a thread tries to acquire a lock that is in the locked state, it will wait until some thread releases it. (Unless it set the blocking flag false.) If more that one thread is waiting to acquire the lock, it may be blocked longer. I think the doc for threading.Lock is good: http://docs.python.org/lib/lock-objects.html You're correct. I noticed that even though while one thread acquires the lock, the other threads don't respect the lock. In fact they just go ahead and execute the statements within the lock acquire statement. With this behavior, I'm ending up having a partially corrupted zip archive file. def run(request, response, func=copy_first_match): '''Get items from the request Queue, process them with func(), put the results along with the Thread's name into the response Queue. Stop running once an item is None.''' name = threading.currentThread().getName() ziplock = threading.Lock() while 1: item = request.get() if item is None: break (sUrl, sFile, download_size, checksum) = stripper(item) response.put((name, sUrl, sFile, func(cache, sFile, sSourceDir, checksum))) # This will take care of making sure that if downloaded, they are zipped (thread_name, Url, File, exit_status) = responseQueue.get() if exit_status == False: log.verbose(%s not available in local cache %s\n % (File, cache)) if download_from_web(sUrl, sFile, sSourceDir, checksum) != True: log.verbose(%s not downloaded from %s and NA in local cache %s\n\n % (sFile, sUrl, sRepository)) else: # We need this because we can't do join or exists operation on None if cache is None or os.path.exists(os.path.join(cache, sFile)): #INFO: The file is already there. pass else: shutil.copy(sFile, cache) if zip_bool: ziplock.acquire() try: compress_the_file(zip_type_file, sFile, sSourceDir) os.remove(sFile) # Remove it because we don't need the file once it is zipped. finally: ziplock.release() elif exit_status == True: if zip_bool: ziplock.acquire() try: compress_the_file(zip_type_file, sFile, sSourceDir) os.unlink(sFile) finally: ziplock.release() -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Bryan Olson on Saturday 05 Aug 2006 23:56 wrote: You don't want ziplock = threading.Lock() in the body of the function. It creates a new and different lock on every execution. Your threads are all acquiring different locks. To coordinate your threads, they need to be using the same lock. Try moving ziplock = threading.Lock() out of the function, so your code might read, in part: ziplock = threading.Lock() def run(request, response, func=copy_first_match): # And so on... Thanks. That did it. :-) Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Gerhard Fiedler wrote: Rather than downloading and zipping in the same thread, you could run multiple threads like you're doing that only download files, and one zip-it-all-up thread. After downloading a file, the download threads place a message in a queue that indicates the file they have downloaded, and the zip-it-all-up thread takes messages out of that queue, one at a time, and zips the files. I was using this approach earlier. The problem with this approach is too much temporary disk usage. Say I'm downloading 2 GB of data which is a combination of, say 600 files. Now following this approach, I'll have to make sure that I have 4 GB of disk space available on my hard drive. Where as downloading in pieces, adding them to the archive, and then unlinking the downloaded file helps proper utilization of disk resource. Thanks, Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Carl Banks wrote: If you have multiple threads trying to access the same ZIP file at the same time, whether or not they use the same ZipFile object, you'll have trouble. You'd have to change download_from_web to protect against simultaneous use. A simple lock should suffice. Create the lock in the main thread, like so: ziplock = threading.Lock() Thanks. This looks to be the correct way to go. I do have access to all the source code as it is under GPL. Then change the zipping part of download_from_web to acquire and release this lock; do zipfile operations only between them. ziplock.acquire() try: do_all_zipfile_stuff_here() finally: ziplock.release() I hope while one thread has acquired the lock, the other threads (which have done the downloading work and are ready to zip) would wait. If you can't change download_from_web, you might have no choice but to download sequentially. OTOH, if each thread uses a different ZIP file (and a different ZipFile object), you wouldn't have to use a lock. It doesn't sound like you're doing that, though. It shouldn't be a problem if one thread is zipping at the same time another is downloading, unless there's some common data between them for some reason. Thanks, Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Simon Forman wrote: One other question I had, If my user passes the --zip option, download_from_web() internally (when the download is successful) zips the downloaded data to a zip file. Since in case of threading there'll be multiple threads, and say if one of the thread completes 2 seconds before others and is doing the zipping work: What will the other thread, at that moment do, if it completes while the previous thread is doing the zipping work ? The other threads will just take the next request from the Queue and process it. They won't care what the one thread is doing, downloading, zipping, whatever. The thread will be marked as complete only when the function that it executed exits. Right ? download_from_web() internally calls a funtion to zip the file. So it doesn't return before zipping. Till then this thread is not complete and therefore is busy working (i.e. zipping). during the same time if another thread (which is also calling download_from_web) completes the download, the download function will again call the zip code. At that particular situtation, will it wait for the previous thread to complete the zipping and release the file so that it can zip more data to it or will it just panic and quit ? I think this shouldn't be much concern. The d_f_w() calls zip(). And since zip opens the file.zip file in append mode, no matter how many threads access it, it should be okay. Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Simon Forman wrote: The other threads will just take the next request from the Queue and process it. They won't care what the one thread is doing, downloading, zipping, whatever. As I mentioned in my previous post, the other threads will also have to go through the same zip the file if the download was successful method. I implemented it but am seeing some issues. If I use a single thread, all files are zipped to the archive. Obviously this has to work. If threads are 2 or 3 or 4 in numbers, some of the files don't show up in the archive. What would a thread do (which wants to add files to an archive) if it finds that another thread has opened the archive in append mode ? Will it wait, overwrite or just die ? Looking at the way my program is behaving, it looks like at that condition the threads are skipping/dying or something like that. Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Hi, I have this following situation: #INFO: Thread Support # Will require more design thoughts from Queue import Queue from threading import Thread, currentThread NUMTHREADS = variables.options.num_of_threads def run(request, response, func=download_from_web): '''Get items from the request Queue, process them with func(), put the results along with the Thread's name into the response Queue. Stop running once an item is None.''' name = currentThread().getName() while 1: item = request.get() (sUrl, sFile, download_size, checksum) = stripper(item) if item is None: break response.put((name, func(sUrl, sFile, sSourceDir, None))) My download_from_web() returns True or False depending upon whether the download was successful or failed. How can I check that in the above code ? One other question I had, If my user passes the --zip option, download_from_web() internally (when the download is successful) zips the downloaded data to a zip file. Since in case of threading there'll be multiple threads, and say if one of the thread completes 2 seconds before others and is doing the zipping work: What will the other thread, at that moment do, if it completes while the previous thread is doing the zipping work ? Thanks, Ritesh Justin Azoff on Thursday 27 Jul 2006 22:33 wrote: Ritesh Raj Sarraf wrote: [snip] for item in list_items: download_from_web(item) This way, one items is downloaded at a time. I'm planning to implement threads in my application so that multiple items can be downloaded concurrently. I want the thread option to be user-defined. [snip] See my post about the iterthreader module I wrote... http://groups.google.com/group/comp.lang.python/browse_frm/thread/2ef29fae28cf44c1/ for url, result in Threader(download_from_web, list_items): print url, result #... -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Duncan Booth on Thursday 27 Jul 2006 17:17 wrote: What you want is to use a pool of threads so that you can configure how many requests are issued at a time (you don't want to try to issue 100 requests all in parallel). You can communicate with the threads through a Queue. Thanks to both Duncan and Simon for your motivating replies. I had forgotten the truth that the lists are a medium to learn and educate. I hope you won't be very frustrated by my dummy questions. So if the code for a thread looks like: def run(request, response): while 1: item = request.get() if item is None: break response.put(download_from_web(item)) This is where I'm most confused. I have a list, lRawData which has around, say, 100 items which are read from a file. Currently (without threads), I iterate upon each item in the list, split the item and then manipulate and pass it to download_from_web() to do the real work. I'm mainly confused in your run(). The line response.put(download_from_web(item)) is there. But if I use it, how am I going to iterate on my list lRawData. I'm not able to see where and how, run() or the line item = request.get() goes into my list lRawData to get the items. Can you please explain the functioning of the above code block. # your main loop can be something like: requestQueue = Queue() responseQueue = Queue() thread_pool = [ Thread(target=run, args=(requestQueue, responseQueue) for i in range(numthreads)] for t in thread_pool: t.start() This part seems understandable. for item in list_items: requestQueue.put(item) for i in range(len(list_items)): response = responseQueue.get() handle_response(response) I guess these would be clear once I'm clear on the first problem. # and then to shut down the threads when you've finished: for t in thread_pool: requestQueue.put(None) for t in thread_pool: t.join() Thanks a lot for your replies. Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
And people, Is there any documentation on Python Threads or Threads in general. It'd be of great help to really understand. Ritesh Ritesh Raj Sarraf on Thursday 27 Jul 2006 16:37 wrote: Is this the correct way of threading applications ? This is the first time I'd be doing threading. So was looking for comments and suggestions. -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Simon Forman on Thursday 27 Jul 2006 22:47 wrote: def run(request, response, func=dummy_func): ''' Get items from the request Queue, process them with func(), put the results along with the Thread's name into the response Queue. Stop running once an item is None. ''' name = currentThread().getName() while 1: item = request.get() if item is None: break response.put((name, func(item))) Meanwhile, instead of sitting idle and waiting for a reply, I thought of trying to understand the code (the example by Simon). Good part is that I was able to use it. :-) Here's the changed code: from Queue import Queue from threading import Thread, currentThread # Imports for the dummy testing func. from time import sleep from random import random NUMTHREADS = 3 def run(request, response, func=download_from_web): ''' Get items from the request Queue, process them with func(), put the results along with the Thread's name into the response Queue. Stop running once an item is None. ''' name = currentThread().getName() while 1: item = request.get() (sUrl, sFile, download_size, checksum) = stripper(item) if item is None: break response.put((name, func(sUrl, sFile, sSourceDir, None))) # Create two Queues for the requests and responses requestQueue = Queue() responseQueue = Queue() # Pool of NUMTHREADS Threads that run run(). thread_pool = [ Thread( target=run, args=(requestQueue, responseQueue) ) for i in range(NUMTHREADS) ] # Start the threads. for t in thread_pool: t.start() # Queue up the requests. for item in lRawData: requestQueue.put(item) # Shut down the threads after all requests end. # (Put one None sentinel for each thread.) for t in thread_pool: requestQueue.put(None) # Don't end the program prematurely. # # (Note that because Queue.get() is blocking by # default this isn't strictly necessary. But if # you were, say, handling responses in another # thread, you'd want something like this in your # main thread.) for t in thread_pool: t.join() I'd like to put my understanding over here and would be happy if people can correct me at places. So here it goes: Firstly the code initializes the number of threads. Then it moves on to initializing requestQueue() and responseQueue(). Then it moves on to thread_pool, where it realizes that it has to execute the function run(). From NUMTHREADS in the for loop, it knows how many threads it is supposed to execute parallelly. So once the thread_pool is populated, it starts the threads. Actually, it doesn't start the threads. Instead, it puts the threads into the queue. Then the real iteration, about which I was talking in my earlier post, is done. The iteration happens in one go. And requestQueue.put(item) puts all the items from lRawData into the queue of the run(). But there, the run() already known its limitation on the number of threads. No, I think the above statement is wrong. The actual pool about the number of threads is stored by thread_pool. Once its pool (at a time 3 as per this example) is empty, it again requests for more threads using the requestQueue() And in function run(), when the item of lRawData is None, the thread stops. The the cleanup and checks of any remaining threads is done. Is this all correct ? I also do have a couple of questions more which would be related to locks. But I'd post them once I get done with this part. Thanks, Ritesh -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com Necessity is the mother of invention. Stealing logic from one person is plagiarism, stealing from many is research. The great are those who achieve the impossible, the petty are those who cannot - rrs -- http://mail.python.org/mailman/listinfo/python-list
Thread Question
Hi, I have some basic doubts about thread. I have a list which has items in it which need to be downloaded from the internet. Let's say list is: list_items[] which has 100 items in it. I have a function download_from_web() which does the work of downloading the items from the web. It does error handling in case it gets errors like 404 et cetera. Currently, this is how I'm using it. for item in list_items: download_from_web(item) This way, one items is downloaded at a time. I'm planning to implement threads in my application so that multiple items can be downloaded concurrently. I want the thread option to be user-defined. Looking at the documentation of threads (Core Python Programming), I've noticed that all threads are executed a once. Depending upon what they are doing, some finish early and some later. But I want to implement something like: for item in list_items: for num in thread_args: thread[num].start() thread[num].start() Is this the correct way of threading applications ? This is the first time I'd be doing threading. So was looking for comments and suggestions. Thanks, Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread Question
Duncan, I couldn't make out much from the code. Instead this is what I did. threads = [] nloops = range(len(lRawData)) for i in nloops: (sUrl, sFile, download_size, checksum) = stripper(lRawData[i]) t = threading.Thread(target=download_from_web, args=(sUrl, sFile, sSourceDir, None)) # = pypt_thread(download_from_web, i, stripper(lRawData[i])) threads.append(t) i = 0 join_i = 0 while i nloops: counter = 0 while counter 3: threads[i].start() counter += 1 i += 1 counter = 0 join_i = i - 3 while counter 3: threads[join_i].join() counter += 1 join_i += 1 Is this correct ? Comments!! Ritesh Duncan Booth wrote: Ritesh Raj Sarraf wrote: I'm planning to implement threads in my application so that multiple items can be downloaded concurrently. I want the thread option to be user-defined. Looking at the documentation of threads (Core Python Programming), I've noticed that all threads are executed a once. Depending upon what they are doing, some finish early and some later. But I want to implement something like: for item in list_items: for num in thread_args: thread[num].start() thread[num].start() Is this the correct way of threading applications ? This is the first time I'd be doing threading. So was looking for comments and suggestions. What you want is to use a pool of threads so that you can configure how many requests are issued at a time (you don't want to try to issue 100 requests all in parallel). You can communicate with the threads through a Queue. So if the code for a thread looks like: def run(request, response): while 1: item = request.get() if item is None: break response.put(download_from_web(item)) # your main loop can be something like: requestQueue = Queue() responseQueue = Queue() thread_pool = [ Thread(target=run, args=(requestQueue, responseQueue) for i in range(numthreads)] for t in thread_pool: t.start() for item in list_items: requestQueue.put(item) for i in range(len(list_items)): response = responseQueue.get() handle_response(response) # and then to shut down the threads when you've finished: for t in thread_pool: requestQueue.put(None) for t in thread_pool: t.join() -- http://mail.python.org/mailman/listinfo/python-list
class instance scope
Hi, I have a class defined in a file called foo.py In bar.py I've imported foo.py In bar.py's main function, I instantiate the class as follows: log = foo.log(x, y, z) Now in main I'm able to use log.view(), log.error() et cetera. But when I call the same method from some functions which are in bar.py, it fails giving me the following error: NameError: global name 'log' is not defined 1) I tried lookng into the docs but couldn't find anything on instance scope. 2) How is such situation tackled ? Will I have to instantiate in every function ? Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: class instance scope
Steve Holden wrote: Ritesh Raj Sarraf wrote: But when I call the same method from some functions which are in bar.py, it fails giving me the following error: NameError: global name 'log' is not defined Well, that's preumbaly because your log = foo.log(x, y, z) statement was inside a function, and so the name foo was created in that function's local namespace rather than in the module's global namespace. So if I do the instantiation before calling main(), will it work. Something like: if __name__ == __main__: log = foo.log(x, y, z) main() In this case, will log be global ? But still I get the same error. 1) I tried lookng into the docs but couldn't find anything on instance scope. 2) How is such situation tackled ? Will I have to instantiate in every function ? The best thing to do would be to pass the instance in as an argument to the functions that need to manipulate it. But then how do os, sys, and other modules which are imported, become accessible to all the functions ? I'm a newbie, so please bear with me. Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: class instance scope
log = foo.log(x, y, z) Resulting line is: log = foo.log(x, y, z) global log Making the instance log global makes it accessible to all the functions. Now I have only one question, Is this a correct way to do it ? Or are there better way ? Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: python logging module problem
Vinay Sajip wrote: It's usual to rely on logger levels and to set handler levels for additional refinement of what goes to a particular handler's destination. The problem is that for StreamHandler, logging module logs to sys.stderr. I want to use the logging feature for most of the messages my program displays. So some messages would be going to sys.stdout and some to sys.stderr. So, I'm ended up creating two handlers, one for sys.stdout and the other for sys.stderr. I'd then make INFO level messages go to sys.stdout and DEBUG level messages go to sys.stderr. What do you suggest ?? Is it good doing this way ? Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: python logging module problem
Peter Otten wrote: You can achieve the desired behaviour by adding a custom Filter: import sys import logging logger = logging.getLogger(my_app) logger.setLevel(logging.DEBUG) class LevelFilter(logging.Filter): def __init__(self, level): self.level = level def filter(self, record): return self.level == record.levelno def make_handler(outstream, format, level): handler = logging.StreamHandler(outstream) formatter = logging.Formatter(format) handler.setFormatter(formatter) handler.addFilter(LevelFilter(level)) return handler logger.addHandler(make_handler(sys.stderr, 'STDERR %(levelname)s %(message)s', logging.WARN)) logger.addHandler(make_handler(sys.stdout, 'STDOUT %(levelname)s %(message)s', logging.INFO)) logger.info(the world is flat) logger.warning(take care not to fall off its rim) Not sure whether this is a good idea. Another way might be to use distinct loggers. Peter Thanks. This looks similar to what I wanted. I'll try customizing it to my requirements and see if this helps. Thanks, Ritesh -- http://mail.python.org/mailman/listinfo/python-list
python logging module problem
import os, sys, logging
logger = logging.getLogger(my_app)
conerr = logging.StreamHandler(sys.stderr)
conerr.setLevel(logging.DEBUG)
conerr_formatter = logging.Formatter('%(levelname)s %(message)s')
conerr.setFormatter(conerr_formatter)
console = logging.StreamHandler(sys.stdout)
console.setLevel(logging.INFO)
console_formatter = logging.Formatter('%(message)s')
console.setFormatter(console_formatter)
logger.addHandler(conerr)
logger.addHandler(console)
logger.info(Ritesh Raj Sarraf.\n)
logger.warning(Ricky Raj Sarraf.\n)
Hi,
When I execute the above code, logger.info()'s messages don't get
displayed. And logger.warning()'s messages get displayed twice.
C:\Eclipse\Workspace\Python Funpython log.py
WARNING Ricky Raj Sarraf.
Ricky Raj Sarraf.
Is there something I am doing wrong ?
I basically want to use Python's logging module for my entire program.
I want is something like logger.message() which would contain normal
program messages which shouldbe passed to stdout.
I also want to implement a logger.verbose() handler which would execute
when we enable verbose mode.
Am I doing it the correct way ? Or am I using the wrong tool ? Should
logging be used for it ?
TIA,
Ritesh
--
http://mail.python.org/mailman/listinfo/python-list
Re: python logging module problem
Ritesh Raj Sarraf wrote:
import os, sys, logging
logger = logging.getLogger(my_app)
I tried this code:
import logging, sys
# set up logging to file - see previous section for more details
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(name)-12s %(levelname)-8s
%(message)s',
stream=sys.stderr)
# define a Handler which writes INFO messages or higher to the
sys.stderr
console = logging.StreamHandler(sys.stdout)
console.setLevel(logging.INFO)
# set a format which is simpler for console use
formatter = logging.Formatter('%(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
#logging.getLogger('').addHandler(console)
logging.RootLogger(console)#.addHandler(console)
# Now, we can log to the root logger, or any other logger. First the
root...
logging.info('Jackdaws love my big sphinx of quartz.')
logging.debug('Ritesh raj Sarraf.\n')
With this it seems to be working halfway.
logging.debug() works perfect. But logging.info() is inheriting the
settings of logging.debug(). For example it is using logging.debug()'s
formatter while displaying. :-(
Ritesh
--
http://mail.python.org/mailman/listinfo/python-list
compressed file ended before the logical end-of-stream was detected
Hi, The program downloads the files from the internet and compresses them to a single zip archive using compress_the_file(). Upon running syncer() which calls the decompress_the_file(), the first iteration succeeds. But upon second iteration, I get an IOError exception with the message: compressed file ended before the logical end-of-stream was detected Any ideas why this happens ? I don't think it has anything to do with bad packets from the internet. Here's the code. There are two functions: 1) compress_the_file() - This function takes files as an argument to it and put all of them into a zip archive def compress_the_file(zip_file_name, files_to_compress, sSourceDir): '''Condenses all the files into one single file for easy transfer''' try: import zipfile except ImportError: sys.stderr.write(Ai! module not found.\n) try: os.chdir(sSourceDir) except: #TODO: Handle this exception pass try: filename = zipfile.ZipFile(zip_file_name, a) except IOError: #INFO By design zipfile throws an IOError exception when you open # in append mode and the file is not present. filename = zipfile.ZipFile(zip_file_name, w) except: #TODO Handle the exception sys.stderr.write(\nAieee! Some error exception in creating zip file %s\n % (zip_file_name)) sys.exit(1) filename.write(files_to_compress, files_to_compress, zipfile.ZIP_DEFLATED) filename.close() My actual program iterates over a loop and sends files to this funtion as it downloads from the internet. 2) decompress_the_file() - This function takes archive file as an argument and decompresses them. The above function decompress_the_file() is called by function syncer() syncer(): This function tells decompress_the_file() what type file it is and at which path to decompress. def syncer(install_file_path, target_path, type=None): '''Syncer does the work of syncing the downloaded files. It syncs install_file_path which could be a valid file path or a zip archive to target_path''' if type == 1: try: import zipfile except ImportError: sys.stderr.write(Ai! Module zipfile not found.\n) sys.exit(1) file = zipfile.ZipFile(install_file_path, r) for filename in file.namelist(): try: import file_magic except ImportError: sys.stderr.write(Ai! Module file_magic not found.\n) sys.exit(1) data = open(filename, wb) data.write(file.read(filename)) data.close() #data = file.read(filename) if file_magic.file(filename) == application/x-bzip2: decompress_the_file(os.path.abspath(filename), target_path, filename, 1) elif file_magic.file(filename) == PGP armored data: try: shutil.copy(filename, target_path) sys.stdout.write(%s file synced.\n % (filename)) except shutil.Error: sys.stderr.write(%s is already present.\n % (filename)) -- http://mail.python.org/mailman/listinfo/python-list
optparse multiple arguments
Hi, I'm having some minor problems with optparse. I'm just worried that someone shouldn't say that multiple argument feature isn't implemented in optpartse. How to tackle multiple arguments to an option ? As far as I dug, I've found that 1 arguments are being ignored. parser.add_option(, --my-option, dest=my_option, action=store, type=string) Now if someone uses it as: ./foo --my-option a b c I want somehow to store all the three arguments but my_option stores only a while ignoring b and c. Any help? Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: optparse multiple arguments
Ritesh Raj Sarraf wrote: Hi, I'm having some minor problems with optparse. I'm just worried that someone shouldn't say that multiple argument feature isn't implemented in optpartse. How to tackle multiple arguments to an option ? As far as I dug, I've found that 1 arguments are being ignored. parser.add_option(, --my-option, dest=my_option, action=store, type=string) Now if someone uses it as: ./foo --my-option a b c I want somehow to store all the three arguments but my_option stores only a while ignoring b and c. I just noticed that the args variable is holding values b and c. the args variables comes from: (options, args) = parser.parse_args() I guess I only need to figure out now is why args isn't storing argument a also... Ritesh -- http://mail.python.org/mailman/listinfo/python-list
FIXED: Re: optparse multiple arguments
Ritesh Raj Sarraf wrote: I just noticed that the args variable is holding values b and c. the args variables comes from: (options, args) = parser.parse_args() I guess I only need to figure out now is why args isn't storing argument a also... Ritesh I fixed it, I guess. parser.add_option(, --my-option, dest=my_option, action=store_true) sets my_option to True and the arguments are all stored in the list args. :-) Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: FIXED: Re: optparse multiple arguments
Simon Percivall wrote: It might do you good to read the documentation instead of blindly experimenting. Anyway, parser.add_option(, --my-option, nargs=3) http://docs.python.org/lib/optparse-standard-option-actions.html That won't help because by design of my program, I can't limit the number of arguments a user can pass to it. Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: FIXED: Re: optparse multiple arguments
Fredrik Lundh wrote: Ritesh Raj Sarraf wrote: http://docs.python.org/lib/optparse-standard-option-actions.html That won't help because by design of my program, I can't limit the number of arguments a user can pass to it. do you want options, arguments, or are you just somewhat confused ? /F I'm not sure about that. My program will accept user input. The user can input multiple arguments. maybe 1 or maybe 100, that's not definite. I do need option because this particular (multiple) argument criteria is for one single option. Thanks, Ritesh -- http://mail.python.org/mailman/listinfo/python-list
zipfile module doesn't allow append
Hi, I've got a problem here. def compress_the_file(zip_file_name, files_to_compress, sSourceDir): Condenses all the files into one single file for easy transfer try: import zipfile except ImportError: sys.stderr.write(Ai! module not found.\n) try: os.chdir(sSourceDir) except: #TODO: Handle this exception pass filename = zipfile.ZipFile(zip_file_name, a) #try: #filename = zipfile.ZipFile(zip_file_name, a) #except: ##TODO Handle the exception #sys.stderr.write(\nAieee! Some error exception in creating zip file %s\n % (zip_file_name)) #sys.exit(1) filename.write(files_to_compress, files_to_compress, zipfile.ZIP_DEFLATED) filename.close() The line filename = zipfile.ZipFile(zip_file_name, a) throws an exception if the given filename is not present already. Shouldn't it create a file (in case one is not there) since it is append mode ?? Ritesh -- http://mail.python.org/mailman/listinfo/python-list
determining file type
Hi, I have a funtion named unzipper() which does the work of unzipping the files. Is there a way I can identify what is the type of the file which'll be passed to unzipper(). If yes, I'll be able to make out if it's a zip file or a tar or a bz2 file. Thanks, Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: determining file type
But isn't there any library function ?
Something like
XX.filetype(name)
Directory
File-Tar
File-Zip
File-MPEG
Ritesh
Maric Michaud wrote:
Le Mercredi 14 Juin 2006 11:22, Ritesh Raj Sarraf a écrit :
Hi,
I have a funtion named unzipper() which does the work of unzipping the
files.
Is there a way I can identify what is the type of the file which'll be
passed to unzipper().
If yes, I'll be able to make out if it's a zip file or a tar or a bz2
file.
Under Linux you can use the file command in a pipe.
In all cases, you use the magic repository of the file command directly like
this (I found the '\xff\xd8' for jpeg in /usr/share/file/magic.mime on my
debian)
In [69]: f = file ('samurai_tux.jpg')
In [70]: s = f.read(2)
In [71]: s == '\xff\xd8'
Out[71]: True
Thanks,
Ritesh
--
_
Maric Michaud
_
Aristote - www.aristote.info
3 place des tapis
69004 Lyon
Tel: +33 426 880 097
--
http://mail.python.org/mailman/listinfo/python-list
Re: determining file type
Also,
f = file ('some_file.jpg')
throws an error.
str object is not callable
Ritesh
Maric Michaud wrote:
Le Mercredi 14 Juin 2006 11:22, Ritesh Raj Sarraf a écrit :
Hi,
I have a funtion named unzipper() which does the work of unzipping the
files.
Is there a way I can identify what is the type of the file which'll be
passed to unzipper().
If yes, I'll be able to make out if it's a zip file or a tar or a bz2
file.
Under Linux you can use the file command in a pipe.
In all cases, you use the magic repository of the file command directly like
this (I found the '\xff\xd8' for jpeg in /usr/share/file/magic.mime on my
debian)
In [69]: f = file ('samurai_tux.jpg')
In [70]: s = f.read(2)
In [71]: s == '\xff\xd8'
Out[71]: True
Thanks,
Ritesh
--
_
Maric Michaud
_
Aristote - www.aristote.info
3 place des tapis
69004 Lyon
Tel: +33 426 880 097
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python Debugger / IDE ??
Can you please point some good documents (need not be Python specific) on best practices with writing code this way ? Thanks, Ritesh -- http://mail.python.org/mailman/listinfo/python-list
Re: breaking from loop
Thanks to everyone. It is really the best place and the best people to learn from. Here's what I followed from the discussion: def files(root): for path, folders, files in os.walk(root): for file in files: yield path, file def copy_first_match(repository, filename, dest_dir): # aka walk_tree_copy() for path, file in files(repository): if file == filename: try: shutil.copy(os.path.join(path, file), dest_dir) sys.stdout.write(%s copied from local cache %s. % (file, repository)) except shutil.Error: sys.stdout.write(%s is available in %s. Skipping Copy! % (file, dest_dir)) return True return False All I've added is the exception check because in case file is available in dest_dir, I want to display a message. Since I'm still new and learning will be great if you let me know if this is the proper way or not. Thanks to everyone once again. It's been a very good experience understanding everyone's comments. Regards, rrs -- http://mail.python.org/mailman/listinfo/python-list
breaking from loop
Hi,
Following is the code:
def walk_tree_copy(sRepository, sFile, sSourceDir, bFound = None):
try:
if sRepository is not None:
for name in os.listdir(sRepository):
path = os.path.join(sRepository, name)
if os.path.isdir(path):
walk_tree_copy(path, sFile, sSourceDir, bFound)
elif name.endswith('.foo') or name.endswith('.bar'):
if name == sFile:
try:
shutil.copy(path, sSourceDir)
except IOError, (errno, errstring):
errfunc(errno, errstring)
except shutil.Error:
print name + is available in +
sSourceDir + Skipping Copy!
bFound = True
break
return bFound
except OSError, (errno, strerror):
print errno, strerror
This function allows me to walk into a directory based tree and search
for files ending with .foo and .bar. My requirement is to copy
.foo/.bar.
Say in directory temp=test=test.bar is found. So shutil.copy will
copy it. I want that once the copy is done, it should make bFound =
True and get out.
But since test directory is under temp, work_tree_copy makes two calls
of the same function _but_ break only is able to get out from the inner
call.
Where am I wrong in this code ? Is there a better way to implement it ?
Regards,
rrs
--
http://mail.python.org/mailman/listinfo/python-list
[FIXED] Re: urllib2.urlopen Progress bar
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Calling urllib2.urlopen the below mentioned way fixed my problem. try: os.chdir (sSourceDir) block_size = 4096 i = 0 count = 0 temp = urllib2.urlopen(sUrl) headers = temp.info() size = int(headers['Content-Length']) data = open (sFile, 'wb') while i size: data.write (temp.read(block_size)) i += block_size counter += 1 progressbar.myReportHook(counter, block_size, size) print \n data.close() temp.close() Thanks, Ritesh Ritesh Raj Sarraf on Sunday 15 Jan 2006 12:55 wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, In urllib.urlretrieve I can use reporthook to implement a progress bar. But in urllib2.urlopen I can't. I have to use urllib2.urlopen because it throws exceptions which are important to be handled where as urlretrieve is dumb. Is there a way to implement urllib2.urlopen to enable reporthook ? progress.py code: #!/usr/bin/env python import time, urllib2, urllib class progressBar: def __init__(self, minValue = 0, maxValue = 10, totalWidth=12): self.progBar = [] # This holds the progress bar string self.min = minValue self.max = maxValue self.span = maxValue - minValue self.width = totalWidth self.amount = 0 # When amount == max, we are 100% done self.updateAmount(0) # Build progress bar string def updateAmount(self, newAmount = 0): if newAmount self.min: newAmount = self.min if newAmount self.max: newAmount = self.max self.amount = newAmount # Figure out the new percent done, round to an integer diffFromMin = float(self.amount - self.min) percentDone = (diffFromMin / float(self.span)) * 100.0 percentDone = round(percentDone) percentDone = int(percentDone) # Figure out how many hash bars the percentage should be allFull = self.width - 2 numHashes = (percentDone / 100.0) * allFull numHashes = int(round(numHashes)) # build a progress bar with hashes and spaces self.progBar = [ + '#'*numHashes + ' '*(allFull-numHashes) + ] # figure out where to put the percentage, roughly centered percentPlace = (len(self.progBar) / 2) - len(str(percentDone)) percentString = str(percentDone) + % # slice the percentage into the bar #self.progBar = self.progBar[0:percentPlace] + percentString + self.progBar[percentPlace+len(percentString):] self.progBar = self.progBar[0:percentPlace] + self.progBar[percentPlace+len(percentString):] + percentString def __str__(self): return str(self.progBar) def myReportHook(count, blockSize, totalSize): import sys global prog if prog == : prog = progressBar(0,totalSize,50) #print count, blockSize, totalSize #prog = progressBar(0, totalSize, 77) prog.updateAmount(count*blockSize) sys.stdout.write (str(prog)) sys.stdout.write (\r) #print count * (blockSize/1024) , kb of , (totalSize/1024) , kb downloaded.\n prog = sFile = new.rpm sUrl=http://localhost/new.rpm; urllib.urlretrieve(sUrl, sFile, reporthook=myReportHook) print \n\n #temp = urllib2.urlopen(sUrl) #lastval = int(temp.headers['Content-Length']) #prog = progressBar(0, lastval, 77) # #for x in range(101): #prog.updateAmount(x) #print prog, \r, time.sleep(0.5) # #data = open(sFile,'wb') #data.write(temp.read()) #data.close() #temp.close() Regards, Ritesh - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Gnupg Key ID: 04F130BC Stealing logic from one person is plagiarism, stealing from many is research. Necessity is the mother of invention. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDyfkM4Rhi6gTxMLwRAlmMAJ9yePv6DvXCcnUb0DYhWSuIonciSACgrxDj aKSGkfg8I1T/q3r6zWLTXmw= =uomB -END PGP SIGNATURE- - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Stealing logic from one person is plagiarism, stealing from many is research. Necessity is the mother of invention. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDyicF4Rhi6gTxMLwRAldhAJ9cdnma+FerMVxe/EAG+eokL1POdQCgr5A3 ZyMQFN96Yrkp5frgHPYUnfU= =Y9wS -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
urllib2.urlopen Progress bar
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, In urllib.urlretrieve I can use reporthook to implement a progress bar. But in urllib2.urlopen I can't. I have to use urllib2.urlopen because it throws exceptions which are important to be handled where as urlretrieve is dumb. Is there a way to implement urllib2.urlopen to enable reporthook ? progress.py code: #!/usr/bin/env python import time, urllib2, urllib class progressBar: def __init__(self, minValue = 0, maxValue = 10, totalWidth=12): self.progBar = [] # This holds the progress bar string self.min = minValue self.max = maxValue self.span = maxValue - minValue self.width = totalWidth self.amount = 0 # When amount == max, we are 100% done self.updateAmount(0) # Build progress bar string def updateAmount(self, newAmount = 0): if newAmount self.min: newAmount = self.min if newAmount self.max: newAmount = self.max self.amount = newAmount # Figure out the new percent done, round to an integer diffFromMin = float(self.amount - self.min) percentDone = (diffFromMin / float(self.span)) * 100.0 percentDone = round(percentDone) percentDone = int(percentDone) # Figure out how many hash bars the percentage should be allFull = self.width - 2 numHashes = (percentDone / 100.0) * allFull numHashes = int(round(numHashes)) # build a progress bar with hashes and spaces self.progBar = [ + '#'*numHashes + ' '*(allFull-numHashes) + ] # figure out where to put the percentage, roughly centered percentPlace = (len(self.progBar) / 2) - len(str(percentDone)) percentString = str(percentDone) + % # slice the percentage into the bar #self.progBar = self.progBar[0:percentPlace] + percentString + self.progBar[percentPlace+len(percentString):] self.progBar = self.progBar[0:percentPlace] + self.progBar[percentPlace+len(percentString):] + percentString def __str__(self): return str(self.progBar) def myReportHook(count, blockSize, totalSize): import sys global prog if prog == : prog = progressBar(0,totalSize,50) #print count, blockSize, totalSize #prog = progressBar(0, totalSize, 77) prog.updateAmount(count*blockSize) sys.stdout.write (str(prog)) sys.stdout.write (\r) #print count * (blockSize/1024) , kb of , (totalSize/1024) , kb downloaded.\n prog = sFile = new.rpm sUrl=http://localhost/new.rpm; urllib.urlretrieve(sUrl, sFile, reporthook=myReportHook) print \n\n #temp = urllib2.urlopen(sUrl) #lastval = int(temp.headers['Content-Length']) #prog = progressBar(0, lastval, 77) # #for x in range(101): #prog.updateAmount(x) #print prog, \r, time.sleep(0.5) # #data = open(sFile,'wb') #data.write(temp.read()) #data.close() #temp.close() Regards, Ritesh - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Gnupg Key ID: 04F130BC Stealing logic from one person is plagiarism, stealing from many is research. Necessity is the mother of invention. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDyfkM4Rhi6gTxMLwRAlmMAJ9yePv6DvXCcnUb0DYhWSuIonciSACgrxDj aKSGkfg8I1T/q3r6zWLTXmw= =uomB -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: option argument length
On Tue, 6 Dec 2005, Peter Otten wrote: Ritesh Raj Sarraf wrote: I'm using this for option arguments which are mutually inclusive. But I want the user to pass atleast one option argument for the program to function properly. For example, I have an option --fetch-update which requires a file foo to check what it has to fetch. If the file is provided as an argument, it uses it, else I add a parser.set_defaults(foo) which makes the program to look for it in the current working directory. WHen the program see the --fetch-update option, it should execute the required code. Now how do I check if at least one option has been passed at the command-line ? I have multiple options but I have parser.set_defaults() for each of them. I'm sorry I don't understand your example. Wouldn't you need at least two options to demonstrate mutually inclusive options? The set_default() method seems to accept only keyword arguments -- but even it were used correctly I'm still unclear why you would need it at all. Perhaps you can post a few sample invocations (both correct and illegal) of your script together with a description (in English, not code) of how the script should react? Peter, puzzled try: version = 0.6b reldate = 03/10/2005 copyright = (C) 2005 Ritesh Raj Sarraf - RESEARCHUT (http://www.researchut.com/) #FIXME: Option Parsing # There's a flaw with either optparse or I'm not well understood with it # Presently you need to provide all the arguments to it to work. # No less, no more. This needs to be converted to getopt sometime. #parser = OptionParser() #parser = optparse.OptionParser() parser = optparse.OptionParser(usage=%prog [OPTION1, OPTION2, ...], version=%prog + version) parser.add_option(-d,--download-dir, dest=download_dir, help=Root directory path to save the downloaded files, action=store, type=string) parser.set_defaults(download_dir=foo) parser.add_option(-s,--cache-dir, dest=cache_dir, help=Root directory path where the pre-downloaded files will be searched. If not, give a period '.',action=store, type=string, metavar=.) parser.set_defaults(cache_dir=.) #parser.set_defaults(cache_dir=.) #parser.add_option(-u,--uris, dest=uris_file, help=Full path of the uris file which contains the main database of files to be downloaded,action=store, type=string) # We'll have additional options # --set-update - This will extract the list of uris which need to be fetched # --fetch-update - This will fetch the list of uris which need for update. # --install-update - This will install the fetched database files # The same will happen for upgradation. # --set-upgrade - This will extract the list of uris which need to be fetched # --fetch-upgrade - This will fetch the list of uris which need for upgrade # --install-upgrade - This will install the fetched database files parser.add_option(,--set-update, dest=set_update, help=Extract the list of uris which need to be fetched for _updation_, action=store, type=string, metavar=foo) parser.set_defaults(set_update=foo) parser.add_option(,--fetch-update, dest=fetch_update, help=Fetch the list of uris which are needed for _updation_., action=store, type=string, metavar=foo) parser.set_defaults(fetch_update=foo) parser.add_option(,--install-update, dest=install_update, help=Install the fetched database files , action=store, type=string, metavar=foo.zip) parser.set_defaults(install_update=foo.zip) parser.add_option(,--set-upgrade, dest=set_upgrade, help=Extract the list of uris which need to be fetched , action=store, type=string, metavar=foo.dat) parser.set_defaults(set_upgrade=foo.dat) parser.add_option(,--fetch-upgrade, dest=fetch_upgrade, help=Fetch the list of uris which are needed , action=store, type=string, metavar=foo.dat) parser.set_defaults(fetch_upgrade=foo.dat) parser.add_option(,--install-upgrade, dest=install_upgrade, help=Install the fetched packages , action=store, type=string, metavar=foo-fetched.zip) parser.set_defaults(install_ugprade=foofetched.zip) (options, args) = parser.parse_args() #parser.check_required(-d, -s, -u) #if len(arguments) != 2: #parser.error(Err! Incorrect number of arguments. Exiting) if len(options) != 1 or len(options) 2: print len(args) parser.error(No arguments were passed\n) sys.exit(1) elif not options.set_upgrade and options.upgrade_type: parser.error(Only options --set-upgrade and --upgrade-type are mutually inclusive\n) sys.exit(1) Thanks, rrs -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Stealing logic from one person
Re: option argument length
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Peter, Peter Otten on Wednesday December 7 2005 21:25 wrote: This can be simplified to parser.add_option(-d, --download-dir, default=foo, help=Root directory path to save the downloaded files) which seems to be the reason why I've never seen the set_defaults() call before. As per python docs, it's mentioned that default= has been deprecated. That's why I changed to parser.set_defaults() if len(options) != 1 or len(options) 2: It doesn't matter much as it won't work anyway, but len(options) 2 implies len(options) != 1, so if len(options) != 1: #... Yes, you're right. Sorry. would suffice here. But will len(options) give a meaningful output. options is an instance. It gives a Value Error. Now to the actual problem: I think you didn't understand my previous question. I cannot infer from your non-working code what it actually should do. I asked for examples of how your script would be used. E. g, assuming the code above is in a file called sarraf.py, what should the following invocations ./sarraf.py --fetch-update bar ./sarraf.py --fetch-update bar --the-mutually-inclusive-option baz do? Would the first terminate with an error message that another option must also be given? Would it use the default? Would the second be accepted? Try to describe it as simple and clear as possible. Imagine you were talking to someone who has never written a line of code. ./sarraf.py --fetch-update /bar If the user gives the /bar argument, the program should save the downloaded files to /bar. But I'm assuming that the user could be dumb or too lazy, in which case --fetch-udpate should use the parser.set_defaults value i.e. /foo ./sarraf.py --set-upgrade foo.dat --upgrade-type minimal set-upgrade will again write data to foo.dat. If the user doesn't pass it as an arguemnt it should take the defaults (again whatever is there in parser.set_defaults). This will be inclusive with the --upgrade-type option because the user will have a choice of selecting what kind of upgrade he'd like to do, minimal or full or blah. For this I think this should be enough: if not options.set_upgrade and options.upgrade_type: parser.error(They are mutually inclusive options) But my main concern is what if the user doesn't pass any arguemtns. Every option I have has a default value. So I want to check what the user has passed. But unfortunately the args variable has 0 as its value always. Is my way (up till now) of using optparse logically incorrect or improper ? Regards, rrs - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Stealing logics from one person is plagiarism, stealing from many is research. Necessity is the mother of invention. Note: Please CC me. I'm not subscribed to the list -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDlzC84Rhi6gTxMLwRAtCBAJ9z5zDQ8oyx8Jy/rLe9JrwLII3xtACfTaEV VhMmj8OD+/p+yN/8wF6xe+8= =atC6 -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: option argument length
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Marc 'BlackJack' Rintsch on Monday December 5 2005 03:24 wrote: In [EMAIL PROTECTED], Ritesh Raj Sarraf wrote: My program uses mostly option arguments hence my len(args) value is always zero. I need to check if the user has passed the correct number of option arguments. Something like: (options,args) = parser.parse_args() len(options) != 1 or len(options) 2: print Incorrect number of arguments passed. How do I accomplish it ? Just insert an ``if`` in front of the condition and end the program with ``sys.exit()`` after the message. Ciao, Marc 'BlackJack' Rintsch This won't help because options is an instance. Regards, rrs - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Stealing logics from one person is plagiarism, stealing from many is research. Necessity is the mother of invention. Note: Please CC me. I'm not subscribed to the list -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDlIbH4Rhi6gTxMLwRAlgSAJ0Y3TT9eCBgrck5N2Y9YjOTZMxUgwCcDeO5 qqgzY6rz2E4YKvurnlHL0nQ= =hlZO -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: option argument length
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Peter Otten on Monday December 5 2005 03:34 wrote: options, args = parser.parse_args(values=MyValues()) but you should do your users a favour and give them meaningful error messages. I can't conceive how you could achieve this by checking the number of options. Explicit constraint checks like options, args = parser.parse_args() if options.eat_your_cake and options.have_it: parser.error(Sorry, you cannot eat your cake and have it) will increase your script's usability and make it easier to maintain for only a tiny amount of work. I'm using this for option arguments which are mutually inclusive. But I want the user to pass atleast one option argument for the program to function properly. For example, I have an option --fetch-update which requires a file foo to check what it has to fetch. If the file is provided as an argument, it uses it, else I add a parser.set_defaults(foo) which makes the program to look for it in the current working directory. WHen the program see the --fetch-update option, it should execute the required code. Now how do I check if at least one option has been passed at the command-line ? I have multiple options but I have parser.set_defaults() for each of them. Regards, rrs - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Stealing logics from one person is plagiarism, stealing from many is research. Necessity is the mother of invention. Note: Please CC me. I'm not subscribed to the list -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDlIhF4Rhi6gTxMLwRAsLTAJ9ydjppGrFVbH2kL00vL00HgtrxqQCghZBq pC0/1HftWE+9Eipx6vF+3Bo= =Ay8m -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
option argument length
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, I'm using optparse module to parse all options and arguments. My program uses mostly option arguments hence my len(args) value is always zero. I need to check if the user has passed the correct number of option arguments. Something like: (options,args) = parser.parse_args() len(options) != 1 or len(options) 2: print Incorrect number of arguments passed. How do I accomplish it ? Regards, rrs - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Stealing logics from one person is plagiarism, stealing from many is research. Necessity is the mother of invention. Note: Please CC me. I'm not subscribed to the list -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDk1Nh4Rhi6gTxMLwRApx0AJ9XHlWFU1J0NdN02gtvimogUSgDkACgmkOO 2pX8ocoC7pot1a8R4u2BWrY= =piNo -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: urllib.urlretireve problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Diez B. Roggisch wrote: You could for instance try and see what kind of result you got using the unix file command - it will tell you that you received a html file, not a deb. Or check the mimetype returned - its text/html in the error case of yours, and most probably something like application/octet-stream otherwise. Using the unix file command is not possible at all. The whole goal of the program is to help people get their packages downloaded from some other (high speed) machine which could be running Windows/Mac OSX/Linux et cetera. That is why I'm sticking strictly to python libraries. The second suggestion sounds good. I'll look into that. Thanks, rrs - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Gnupg Key ID: 04F130BC Stealing logic from one person is plagiarism, stealing from many is research. -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) iD8DBQFCTDhV4Rhi6gTxMLwRAi2BAJ4zp7IsQNMZ1zqpF/hGUAjUyYwKigCeKaqO FbGuuFOIHawZ8y/ICf87wOI= =btA5 -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: urllib.urlretireve problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Larry Bates wrote: I noticed you hadn't gotten a reply. When I execute this it put's the following in the retrieved file: !DOCTYPE HTML PUBLIC -//IETF//DTD HTML 2.0//EN HTMLHEAD TITLE404 Not Found/TITLE /HEADBODY H1Not Found/H1 The requested URL /pool/updates/main/p/perl/libparl5.6_5.6.1-8.9_i386.deb was no t found on this server.P /BODY/HTML You will probably need to use something else to first determine if the URL actually exists. I'm happy that at least someone responded as this was my first post to the python mailing list. I'm coding a program for offline package management. The link that I provided could be obsolete by newer packages. That is where my problem is. I wanted to know how to raise an exception here so that depending on the type of exception I could make my program function. For example, for Temporary Name Resolution Failure, python raises an exception which I've handled well. The problem lies with obsolete urls where no exception is raised and I end up having a 404 error page as my data. Can we have an exception for that ? Or can we have the exit status of urllib.urlretrieve to know if it downloaded the desired file. I think my problem is fixable in urllib.urlopen, I just find urllib.urlretrieve more convenient and want to know if it can be done with it. Thanks for responding. rrs - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Gnupg Key ID: 04F130BC Stealing logic from one person is plagiarism, stealing from many is research. -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) iD8DBQFCSuYS4Rhi6gTxMLwRAu0FAJ9R0s4TyB7zHcvDFTflOp2joVkErQCfU4vG 8U0Ah5WTdTQHKRkmPsZsHdE= =OMub -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
urllib.urlretireve problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello Everybody, I've got a small problem with urlretrieve. Even passing a bad url to urlretrieve doesn't raise an exception. Or does it? If Yes, What exception is it ? And how do I use it in my program ? I've searched a lot but haven't found anything helping. Example: try: urllib.urlretrieve(http://security.debian.org/pool/updates/main/p/perl/libparl5.6_5.6.1-8.9_i386.deb;) except IOError, X: DoSomething(X) except OSError, X: DoSomething(X) urllib.urlretrieve doesn't raise an exception even though there is no package named libparl5.6 Please Help! rrs - -- Ritesh Raj Sarraf RESEARCHUT -- http://www.researchut.com Gnupg Key ID: 04F130BC Stealing logic from one person is plagiarism, stealing from many is research. -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) iD8DBQFCRcCk4Rhi6gTxMLwRAlb2AJ0fB3V5ZpwdAiCxfl/rGBWU92YBEACdFYIJ 8bGZMJ5nuKAqvjO0KEAylUg= =eaHC -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
