[issue40363] shlex.quote applied to a glob pattern disables glob matching
New submission from Jack Orenstein : I am using shlex.quote to pass filenames to a shell command, e.g. ls. The problem is that glob patterns, when quoted, no longer operate as glob patterns. So, for example, executing this: ls 'var/log/syslog*' results in this output: ls: cannot access '/var/log/syslog*': No such file or directory The documentation for shlex.quote says: "Return a shell-escaped version of the string s. The returned value is a string that can safely be used as one token in a shell command line, for cases where you cannot use a list." I believe that quoting needs to preserve glob semantics to fulfill the documented goal. For example, it seems to me that quoting as follows would be better: ls '/var/log/syslog'* -- components: Library (Lib) messages: 367010 nosy: geophile priority: normal severity: normal status: open title: shlex.quote applied to a glob pattern disables glob matching versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue40363> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39333] argparse should offer an alternative to SystemExit in case a parse fails
Jack Orenstein added the comment: Yes! I didn't know about that method, thank you. -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue39333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39333] argparse should offer an alternative to SystemExit in case a parse fails
New submission from Jack Orenstein : If parse_args fails, SystemExit is raised, carrying an exit code of 2, and the help message is printed. For an embedded usage of argparse, this behavior is undesirable. I am writing an interactive console application, using argparse to parse input. When a parse fails, I would like to print an error message and continue, not terminate the program. Currently, I need to catch SystemExit to be able to do this, which has obvious problems, (e.g., what if something other that argparse is causing the exit?) I'd like to see some way to specify alternative behavior, e.g. raise an exception of a given type. -- components: Library (Lib) messages: 359991 nosy: geophile priority: normal severity: normal status: open title: argparse should offer an alternative to SystemExit in case a parse fails type: enhancement ___ Python tracker <https://bugs.python.org/issue39333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: how to remove 50000 elements from a 100000 list?
On May 5, 2006, at 9:36 AM, Ju Hui wrote: a=range(10) b=range(5) for x in b: ... a.remove(x) ... it will very slowly. Shall I change to another data structure and choos a better arithmetic? any suggestion is welcome. If removal is an O(n) operation, then removing 1/2 the list will take O(n**2), which you don't want. You'd be better off with the contents of a being in a hash table (O(1) removal in practice) or a balanced tree (O(log n) removal). Another possibility: If the a and b lists are ordered in the same way, then you could walk through the lists in order using a merge procedure, generating a new list as you go. After ruling out slow data structures and algorithms, you'll almost certainly be better off using something built in to Python rather than coding your own data structure in Python. Jack Orenstein -- http://mail.python.org/mailman/listinfo/python-list
ANN: Object Shell -- osh 0.6
In a Unix shell, you can run a series of commands, connecting the output of one command with the input to the next command using pipes. The information passed from one command to the next is normally organized into lines of text. Tools such as awk and sed are designed to extract and transform information from these lines. It would be much simpler if the data passed between commands was organized into objects. Osh is based on this idea. Because osh is based on Python, it is Python objects that are passed between commands, and Python functions are used to manipulate objects. Osh is licensed under the GPL. For more information and for downloads: http://geophile.com/osh. Jack Orenstein -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations.html
threading.Thread vs. signal.signal
I'd like to create a program that invokes a function once a second, and terminates when the user types ctrl-c. So I created a signal handler, created a threading.Thread which does the invocation every second, and started the thread. The signal handler seems to be ineffective. Any idea what I'm doing wrong? This is on Fedora FC4 and Python 2.4.1. The code appears below. If I do the while ... sleep in the main thread, then the signal handler works as expected. (This isn't really a satisfactory implementation because the function called every second might take a significant fraction of a second to execute.) Jack Orenstein import sys import signal import threading import datetime import time class metronome(threading.Thread): def __init__(self, interval, function): threading.Thread.__init__(self) self.interval = interval self.function = function self.done = False def cancel(self): print ' cancel' self.done = True def run(self): while not self.done: time.sleep(self.interval) if self.done: print ' break!' break else: self.function() def ctrl_c_handler(signal, frame): print ' ctrl c' global t t.cancel() sys.stdout.close() sys.stderr.close() sys.exit(0) signal.signal(signal.SIGINT, ctrl_c_handler) def hello(): print datetime.datetime.now() t = metronome(1, hello) t.start() -- http://mail.python.org/mailman/listinfo/python-list
distutils setup ignoring scripts
I'm using Python 2.2 on RH9. I have a set of Python modules organized into a root package and one other package named foobar. setup.py looks like this: from distutils.core import setup setup( name = 'foobar', version = '0.3', description = 'Foo Bar', author = 'Jack Orenstein', author_email = '[EMAIL PROTECTED]', packages = ['', 'xyz'], scripts = ['bin/foobar'] ) The resulting package has everything in the specified directories, but does not include the script. I've tried making the path bin/foobar absolute, but that doesn't help. I've googled for known bugs of this sort but have come up emtpy. (The first line of bin/foobar is #!/usr/bin/python.) I've also tried using DISTUTIL_DEBUG, which has been uninformative, (e.g. no mention of bin/foobar at all). Can anyone see what I'm doing wrong? Jack Orenstein -- http://mail.python.org/mailman/listinfo/python-list
Re: Threading and consuming output from processes
I asked: I am developing a Python program that submits a command to each node of a cluster and consumes the stdout and stderr from each. I want all the processes to run in parallel, so I start a thread for each node. There could be a lot of output from a node, so I have a thread reading each stream, for a total of three threads per node. (I could probably reduce to two threads per node by having the process thread handle stdout or stderr.) Simon Wittber said: In the past, I have used the select module to manage asynchronous IO operations. I pass the select.select function a list of file-like objects, and it returns a list of file-like objects which are ready for reading and writing. Donn Cave said: As I see another followup has already mentioned, the classic pre threads solution to multiple I/O sources is the select(2) function, ... Thanks for your replies. The streams that I need to read contain pickled data. The select call returns files that have available input, and I can use read(file_descriptor, max) to read some of the input data. But then how can I convert the bytes just read into a stream for unpickling? I somehow need to take the bytes arriving for a given file descriptor and buffer them until the unpickler has enough data to return a complete unpickled object. (It would be nice to do this without copying the bytes from one place to another, but I don't even see how do solve the problem with copying.) Jack -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread scheduling
Peter Hansen wrote: Jack Orenstein wrote: I am using Python 2.2.2 on RH9, and just starting to work with Python threads. Is this also the first time you've worked with threads in general, or do you have much experience with them in other situations? Yes, I've used threading in Java. You've got two shared global variables, done and counter. Each of these is modified in a manner that is not thread-safe. I don't know if counter is causing trouble, but it seems likely that done is. I understand that. As I said in my posting, The counter is incremented without locking -- I expect to see a final count of less than THREADS * COUNT. This is a test case, and I threw out more and more code, including synchronization around counter and done, until it got as simple as possible and still showed the problem. Basically, the statement done += 1 is equivalent to the statement done = done + 1 which, in Python or most other languages is not thread-safe. The done + 1 part is evaluated separately from the assignment, so it's possible that two threads will be executing the done + 1 part at the same time and that the following assignment of one thread will be overwritten immediately by the assignment in the next thread, but with a value that is now one less than what you really wanted. Understood. I was counting on this being unlikely for my test case. I realize this isn't something to rely on in real software. If you really want to increment globals from the thread, you should look into locks. Using the threading module (as is generally recommended, instead of using thread), you would use threading.Lock(). As my note said, I did start with the threading module. And variables updated by different threads were protected by threading.Condition variables. As I analyzed my test cases, and threading.py, I started suspecting thread scheduling. I then wrote the test case in my email, which does not rely on the threading module at all. The point of the test is not to maintain counter -- it's to show that sometimes even after one thread completes, the other thread never is scheduled again. This seems wrong. Try running the code, and let me see if you see this behavior. If you'd like, replace this: counter += 1 by this: time.sleep(0.01 * id) You should see the same problem. So that removes counter from the picture. And the two increments of done (one by each thread) are still almost certainly not going to coincide and cause a problem. Also, if you look at the output from the code on a hang, you will see that 'thread X: leaving' only prints once. This has nothing to do with what happens with the done variable. Jack -- http://mail.python.org/mailman/listinfo/python-list
Re: Thread scheduling
On my machines (one Py2.4 on WinXP, one Py2.3.4 on RH9.0) I don't see this behaviour. Across about fifty runs each. Thanks for trying this. One thing you might try is experimenting with sys.setcheckinterval(), just to see what effect it might have, if any. That does seem to have an impact. At 0, the problem was completely reproducible. At 100, I couldn't get it to occur. It's also possible there were some threading bugs in Py2.2 under Linux. Maybe you could repeat the test with a more recent version and see if you get different behaviour. (Not that that proves anything conclusively, but at least it might be a good solution for your immediate problem.) 2.3 (on the same machine) does seem better, even with setcheckinterval(0). Thanks for your suggestions. Can anyone with knowledge of Python internals comment on these results? (Look earlier in the thread for details. But basically, a very simple program with the thread module, running two threads, shows that on occasion, one thread finishes and the other never runs again. python2.3 seems better, as does python2.2 with sys.setcheckinterval(100).) Jack -- http://mail.python.org/mailman/listinfo/python-list
Threading and consuming output from processes
I am developing a Python program that submits a command to each node of a cluster and consumes the stdout and stderr from each. I want all the processes to run in parallel, so I start a thread for each node. There could be a lot of output from a node, so I have a thread reading each stream, for a total of three threads per node. (I could probably reduce to two threads per node by having the process thread handle stdout or stderr.) I've developed some code and have run into problems using the threading module, and have questions at various levels of detail. 1) How should I solve this problem? I'm an experienced Java programmer but new to Python, so my solution looks very Java-like (hence the use of the threading module). Any advice on the right way to approach the problem in Python would be useful. 2) How many active Python threads is it reasonable to have at one time? Our clusters have up to 50 nodes -- is 100-150 threads known to work? (I'm using Python 2.2.2 on RedHat 9.) 3) I've run into a number of problems with the threading module. My program seems to work about 90% of the time. The remaining 10%, it looks like notify or notifyAll don't wake up waiting threads; or I find some other problem that makes me wonder about the stability of the threading module. I can post details on the problems I'm seeing, but I thought it would be good to get general feedback first. (Googling doesn't turn up any signs of trouble.) Thanks. Jack Orenstein -- http://mail.python.org/mailman/listinfo/python-list
