Re: bug with os.rename in 2.4.1?
t123 wrote: > It's running on solaris 9. Here is some of the code. It's actually > at the beginning of the job. The files are ftp'd over. The first > thing that happens is that the files get renamed before any processing > of the file. And when it fails, it always fails at the first file, > comm.dat. What I can't understand is why the inconsistent behavior. > > try: > if os.path.exists(paths.xferin_dir+'/COMM.DAT'): >os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/ This code is inherently racy... What if two copies of your code started simultaneously? They might both run the os.path.exists but only one will succeed in the os.rename. You could write instead try: os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/COMM.DAT'+'.0') except OSError: pass Which isn't racy. Or if you wanted to be more thorough import errno try: os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/COMM.DAT'+'.0') except OSError, e: if e.errno != errno.ENOENT: raise The traceback should show the exact problem though. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Large data arrays?
Ole Streicher wrote: > Hi Nick, > > Nick Craig-Wood writes: > > mmaps come out of your applications memory space, so out of that 3 GB > > limit. You don't need that much RAM of course but it does use up > > address space. > > Hmm. So I have no chance to use >= 2 of these arrays simultaniously? > > > Sorry don't know very much about numpy, but it occurs to me that you > > could have two copies of your mmapped array, one the transpose of the > > other which would then speed up the two access patterns enormously. > > That would be a solution, but it takes twice the amount of address > space (which seems already to be the limiting factor). In my case (1.6 > GB per array), I could even not use one array. You don't need them mapped at the same time so you could get away with just one copy mapped. Also you can map the array in parts and use dramatically less address space. > Also, I would need to fill two large files at program start: one for > each orientation (row-wise or column-wise). Depending on the input > data (which are also either row-wise or column-wise), the filling of > the array with opposite direction would take a lot of time because of > the inefficiencies. > > For that, using both directions probably would be not a good > solution. What I found is the "Morton layout" which uses a kind of > fractal interleaving and sound not that complicated. It sounds cool! > But I have no idea on how to turn it into a "numpy" style: can I > just extend from numpy.ndarray (or numpy.memmap), and which > functions/methods then need to be overwritten? The best would be > ofcourse that someone already did this before that I could use > without trapping in all these pitfalls which occur when one > implements a very generic algorithm. I'd start by writing a function which took (x, y) in array co-ordinates and transformed that into (z) remapped in the Morton layout. Then instead of accessing array[x][y] you access morton_array[f(x,y)]. That doesn't require any subclassing and is relatively easy to implement. I'd try that and see if it works first! Alternatively you could install a 64bit OS on your machine and use my scheme! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory footpring of python objects
BlueBird wrote: > I have a program that manages several thousands instances of one > object. To reduce memory > consumption, I want of course that specific object to have the > smallest memory footpring possible. > > I have a few ideas that I want to experiment with, like using > __slots__, using a tuple or using a dict. My > question is: how do I know the memory footprint of a given python > object ? I could not find any > builtin functions for this in the documentation. I managed to 1/3 the memory requirement of our program by using __slots__ on the most common class (several hundred thousand instances!). When doing these optimisations I ran a repeatable script and measured the total memory usage using the OS tools (top in my case). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Large data arrays?
Ole Streicher wrote: > for my application, I need to use quite large data arrays > (100.000 x 4000 values) with floating point numbers where I need a fast > row-wise and column-wise access (main case: return a column with the sum > over a number of selected rows, and vice versa). > > I would use the numpy array for that, but they seem to be > memory-resistent. So, one of these arrays would use about 1.6 GB > memory which far too much. So I was thinking about a memory mapped > file for that. As far as I understand, there is one in numpy. > > For this, I have two questions: > > 1. Are the "numpy.memmap" array unlimited in size (resp. only limited > by the maximal file size)? And are they part of the system's memory > limit (~3GB for 32bit systems)? mmaps come out of your applications memory space, so out of that 3 GB limit. You don't need that much RAM of course but it does use up address space. > 2. Since I need row-wise as well as column-wise access, a simple usage > of a big array as memory mapped file will probably lead to a very poor > performance, since one of them would need to read values splattered > around the whole file. Are there any "plug and play" solutions for > that? If not: what would be the best way to solve this problem? > Probably, one needs to use someting like the "Morton layout" for the > data. Would one then build a subclass of memmap (or ndarray?) that > implements this specific layout? How would one do that? (Sorry, I am > still a beginner with respect to python). Sorry don't know very much about numpy, but it occurs to me that you could have two copies of your mmapped array, one the transpose of the other which would then speed up the two access patterns enormously. You needn't mmap the two arrays (files) at the same time either. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: pyflakes, pylint, pychecker - and other tools
Colin J. Williams wrote: > Esmail wrote: > > What is the consensus of the Python community regarding these > > code checkers? > > > > In particular, are the stylistic recommendations that > > pylint makes considered sensible/valid? > > pylint seems a bit heavy handled, a bit > too much PEP 8, which was intended as a > guide, rather than a prescription. I've been very happy with pychecker. I found pylint a bit too fussy (rather like the original C lint!) Note that if you run pychecker from emacs (M-x compile, then "pychecker module_name") you can then click in its output window to go to the correct line of code. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: A Special Thanks
norseman wrote: > I'm one of those that tries to get an outline of the project and then > puts in code as things become clear. Once the basics are working > reasonably I go back and organize the thing for maintainability. Then > finish flushing it out. It is the one stage I dread the most. > > Why not organize it up front? Because I don't always have the whole pie > at the outset. > > In changing to Python I had a bigger learning curve than I realized at > the start. When I finally got my "pieces" accomplishing what I wanted > it became time to start looking at its structure. > > I did the cut and paste into a followable form and ran the basic to > check for the usual errors, omissions and outright flaws. This was a > major reorganization. > > IT RAN FLAWLESSLY THE FIRST TRY! UNBELIEVABLE! (at least for me) Python has a well deserved reputation for being executable pseudo-code. It is the only language I've ever used where when you write the program it is quite likely to work the first time. Python also converted me to using unit tests. If you add unit tests into your methodology above then when you re-organize (or refactor to use the modern jargon) the code you can be 100% sure that you didn't break anything which is a wonderful feeling. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to create a virtual serial port?
Grant Edwards wrote: > On 2009-04-12, JanC wrote: > > Grant Edwards wrote: > > > >> On 2009-04-10, Stuart Davenport wrote: > >> > >>> I am trying to work out if its possible, to create a virtual serial > >>> port with Python? > >> > >> On Linux: no. > > > > I wonder if there is no way to emulate ptys from userspace? > > Didn't I just answer that question? > > On Linux: no. Actually you could do it with an LD_PRELOAD library Intercept open("/dev/ttyS0",...). You'd need to intercept ioctl(), read(), write(), close() etc too. Messy but possible. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: What IDE support python 3.0.1 ?
Well i use netbean is alot better i think and it work with 2.6 and 3.0 --- On Thu, 4/16/09, mousemeat wrote: From: mousemeat Subject: Re: What IDE support python 3.0.1 ? To: python-list@python.org Date: Thursday, April 16, 2009, 4:41 AM Use eclipse with the pydev module. I use python(x,y) which is a big bundle of most of the python stuff you could possibly want (including scientific stuff, but its not mandatory to use it) configured to work together. It uses python 2.5. You can have the best of both worlds. Search for 'from __future__ import' to see how to get 3.0 features into 2.x, minimizing the eventual upgrade hassles. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: interacting with an updatedb generated data file within python
birdsong wrote: > Does anybody have any recommendations on how to interact with the data > file that updatedb generates? I'm running through a file list in > sqlite that I want to check against the file system. updatedb is > pretty optimized for building an index and storing it, but I see no > way to query the db file other than calling locate itself. This would > require me to fork and exec for every single file I want to verify - > I'd be better off doing the stat myself in that case, but I'd really > rather let updatedb build the index for me. Hmm.. There are several different implementations of locate and I'm not sure they all use the same database. On this ubuntu machine the database is only root readable also. $ ls -l /var/lib/mlocate/mlocate.db -rw-r- 1 root mlocate 7013090 2009-04-14 07:54 /var/lib/mlocate/mlocate.db > I searched high and low for any sort of library that is well suited > for reading these data files, but I've found nothing for any language > other than the source for locate and updatedb itself. You can use this to extract the database from the locate database from subprocess import Popen, PIPE from time import time start = time() all_files = set() p = Popen(["locate", "*"], stdout=PIPE) for line in p.stdout: path = line[:-1] all_files.add(path) print "Found", len(all_files), "files in", time()-start, "seconds" This builds a set of all the files on the filesystem and prints Found 314492 files in 1.152987957 seconds on my laptop, using about 19 MB total memory You could easily enough put that into an sqlite table instead of a set(). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Best way to extract from regex in if statement
Paul McGuire wrote: > On Apr 3, 9:26 pm, Paul Rubin <http://phr...@nospam.invalid> wrote: > > bwgoudey writes: > > > elif re.match("^DATASET:\s*(.+) ", line): > > > m=re.match("^DATASET:\s*(.+) ", line) > > > print m.group(1)) > > > > Sometimes I like to make a special class that saves the result: > > > > class Reg(object): # illustrative code, not tested > > def match(self, pattern, line): > > self.result = re.match(pattern, line) > > return self.result > > > I took this a little further, *and* lightly tested it too. > > Since this idiom makes repeated references to the input line, I added > that to the constructor of the matching class. > > By using __call__, I made the created object callable, taking the RE > expression as its lone argument and returning a boolean indicating > match success or failure. The result of the re.match call is saved in > self.matchresult. > > By using __getattr__, the created object proxies for the results of > the re.match call. > > I think the resulting code looks pretty close to the original C or > Perl idiom of cascading "elif (c=re_expr_match("..."))" blocks. > > (I thought about cacheing previously seen REs, or adding support for > compiled REs instead of just strings - after all, this idiom usually > occurs in a loop while iterating of some large body of text. It turns > out that the re module already caches previously compiled REs, so I > left my cacheing out in favor of that already being done in the std > lib.) > > > import re > > class REmatcher(object): > def __init__(self,sourceline): > self.line = sourceline > def __call__(self, regexp): > self.matchresult = re.match(regexp, self.line) > self.success = self.matchresult is not None > return self.success > def __getattr__(self, attr): > return getattr(self.matchresult, attr) That is quite similar to the one I use... """ Matcher class encapsulating a call to re.search for ease of use in conditionals. """ import re class Matcher(object): """ Matcher class m = Matcher() if m.search(r'add (\d+) (\d+)', line): do_add(m[0], m[1]) elif m.search(r'mult (\d+) (\d+)', line): do_mult(m[0], m[1]) elif m.search(r'help (\w+)', line): show_help(m[0]) """ def search(self, r, s): """ Do a regular expression search and return if it matched. """ self.value = re.search(r, s) return self.value def __getitem__(self, n): """ Return n'th matched () item. Note so the first matched item will be matcher[0] """ return self.value.group(n+1) def groups(self): """ Return all the matched () items. """ return self.value.groups() -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: HTML Generation
J Kenneth King wrote: > Stefan Behnel writes: > > See here for another example that uses lxml.html: > > > > http://codespeak.net/lxml/lxmlhtml.html#creating-html-with-the-e-factory > > > > Stefan > > Ah, looks good. Have never used nor finished the example I had given -- > only meant as inspiration. I'm not surprised it has been done by someone > else. Something more like what the OP wanted is HTMLgen. I'm not sure it is maintained any more, but I used it quite a lot a few years ago. Debian still have the source and a package here http://packages.debian.org/sid/python-htmlgen But I think its original website is gone. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
python docs redirect on python.org is old
When I visit ... http://www.python.org/doc/lib/lib.html ... I get redirected to ... http://www.python.org/doc/2.5.2/lib/lib.html ... which seems a bit old. -- Brandon Craig Rhodes bran...@rhodesmill.org http://rhodesmill.org/brandon -- http://mail.python.org/mailman/listinfo/python-list
Re: Sending SMS using python script
There's a Python wrapper to the Skype API here: http://sourceforge.net/projects/skype4py/ On Linux I've used the PyGTK GUI that uses this. It's called SkySentials here: http://www.kolmann.at/philipp/linux/skysentials/ Craig On Apr 3, 6:50 am, "ISF (Computer Scientists without Frontiers, Italy)" wrote: > On 2 Apr, 06:41, guptha wrote: > > > > > hi group, > > my application needs to sendSMSoccasionally to all the clients .Is > > there any library in python that supports in sendingSMS. > > I like to conform few information i gathered in this regard. > > > I can sendSMSby two ways > > > 1. SendingSMSusing Email clients > > 2. Usingsmsgateway to send message(we can implementSMSGateway API > > 's ,provided by vendor and ,sendSMS-- we will be charged > > accordingly ) > > > In case of First approach > > 1. We can make use of libgamil library to sendSMSusing gmail ( I > > ref :http://blog.datasingularity.com/?p=63) > > i suppose sendingsmsthrough gmail is not supported in India > > 2. Can we use Skype4py library, > > > In case of second approach > > 1. Is there any way to sendSMSfor free inside India ,or ,Any freeSMSgateway > > providers in India > > Any information regarding this is appreciable > > Thanks > > A friend from India sent me this hint: > > the following link can be explored further for sending SMS on pyS60 > (python for symbian OS)http://mobilenin.com/pys60/menu.htm > > wkr, > Aldo -- http://mail.python.org/mailman/listinfo/python-list
Re: Beazley on Generators
this is great, thanks... we have used generators to create something akin to a cooperative tasking environment... not to implement multitasking, but to be able to control low level data processing scripts. These scripts, written as generators, yield control to a control loop which then can pause, resume, abort, or change the state of shared context objects which the script uses as it's input and output space. E.g. the control loop can see there is intermediate output which an operator (managing a data reduction pipeline) might want to see. I can see from the first few slide I need to understand this. It already seems clear that there are ways to improve our approach to what we have done, though the overall approach is solid and works well. anyway thanks. -- http://mail.python.org/mailman/listinfo/python-list
Re: double/float precision question
TP wrote: > Hi everybody, > > Try the following python statements: > > >>> "%.40f" % 0.222 > '0.098864108374982606619596' > >>> float( 0.222) > 0.1 > > It seems the first result is the same than the following C program: > > #include > > int main(void) > { > double a = 0.222; > > printf( "%.40f\n", a ); > return 0; > } > # > > My problem is the following: > * the precision "40" (for example) is given by the user, not by the > programmer. > * I want to use the string conversion facility with specifier "e", that > yields number is scientific format; so I cannot apply float() on the result > of "%.40e" % 0.222, I would lost the scientific > format. > > Is there any means to obtain the full C double in Python, or should I limit > the precision given by the user (and used in "%.*e") to the one of a Python > float? Python floats are actually C doubles (as you proved yourself with your little test program). Eg >>> 1.+2.**-52 1.0002 >>> 1.+2.**-53 1.0 Indicating that python floats have about 52 bits of precision, so are definitely what C calls doubles. When you do >>> float( 0.222) 0.1 Python prints as many decimal places as are significant in the answer. This is covered in the FAQ http://www.python.org/doc/faq/general/#why-are-floating-point-calculations-so-inaccurate If you want more precision use the built in decimal module or the third party gmpy module. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: A design problem I met again and again.
一首诗 wrote: > But I think the first step to resolve a problem is to describe it. In > that way, I might find the answer myself :-) That is a great saying! To answer your original question, split your code up into sections that can be tested independently. If you can test code in a isolated way then it belongs in a class / module of its own. If you have a class that is too big, then factor independent classes out of it until it is the right size. That is easier said than done and may require some creativity on your part. It will pay dividends though as the level of abstraction in your program will rise. I've noticed some programmers think in big classes and some think in small classes. Train yourself to do the other thing and your programming will improve greatly! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Ordered Sets
Aahz wrote: > I find the trick of using a Python list to store the doubly-linked > list difficult to understand (as opposed to the usual mechanism of a > node class). I understand why it was done (time and space > efficiency), but I also still feel emotionally that it's somewhat > sick and perverted. I probably would feel more comfortable if the > doubly-linked list were abstracted out and commented. Heck, the > whole thing needs unit tests. Hmmm... Lets compare the two. Here is the length three list $ python Python 2.5.2 (r252:60911, Jan 4 2009, 17:40:26) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> L = [] >>> for i in xrange(100): ... L.append([1,2,3]) ... >>> import os >>> os.getpid() 28134 >>> (from top) 28134 ncw 20 0 58860 53m 1900 S0 2.6 0:02.62 python vs a Node class with __slots__ (for efficiency) $ python Python 2.5.2 (r252:60911, Jan 4 2009, 17:40:26) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class Node(object): ... __slots__ = ["prev", "next", "this"] ... def __init__(self, prev, next, this): ... self.prev = prev ... self.next = next ... self.this = this ... >>> Node(1,2,3) <__main__.Node object at 0xb7e897cc> >>> Node(1,2,3).prev 1 >>> L = [] >>> for i in xrange(100): ... L.append(Node(1,2,3)) ... >>> import os >>> os.getpid() 28203 >>> (from top) 28203 ncw 20 0 43364 38m 1900 S0 1.9 0:04.41 python So the Node class actually takes less memory 38 Mbytes vs 53 Mbytes for the list. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Programming Python 4th Edition?
Esmail wrote: > prueba...@latinmail.com wrote: > > It isn't a introduction to the Python language like "Learning Python", > > it doesn't work as reference like "Python in a Nutshell", it doesn't > > contain short idiomatic code like "Python Cookbook". What you are left > > with is different application domains and how to apply Python to them. > > The book is excellent if you want to do Network, GUI, Databases, etc. > > but poor if you want to learn about Python the core language. The > > title of the book should be changed from "Programming Python" to > > "Applied Python" > > I appreciate you taking the time to post. I agree with what you say. > I guess what appeals to me is the nearly encyclopedic nature of the > book .. and I am curious about scripting with python, so it seems to > have some good material on it (though I think there are newer modules > now available for this). > > It's good to hear what others think about this book, and others > too. I read Programming Python as an experienced programmer and like you I enjoyed the encyclopedic nature of it. So if it appeals to you I'd say go for it! The fact that it doesn't use the latest version of python isn't a problem - python doesn't change very quickly and emphasises backwards compatibility, even for the jump to 3.x. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: New Python Runtime / Unladen Swallow
Adonis wrote: > Came across this article on Ars on a new LLVM (Low Level Virtual > Machine) JIT compiler for Python being built by Google: > > > http://arstechnica.com/open-source/news/2009/03/google-launches-project-to-boost-python-performance-by-5x.ars Interesting article and site. The projct is called Unladen Swallow. (I'd write a link but I can't copy and paste them on my phone!) The idea seems to be to gradually transform python (2.6) into using LLVM as its VM instead of the python VM it uses at the moment. They've already managed to compile python into LLVM (there is a LLVM version of gcc) and speed it up a bit. The next step is to retarget the python byte code generator into producing LLVM bytecode instead. The advantage of targetting LLVM is that it is cross platform and already has JIT and full compilers for many platforms. They plan to fold their work back into CPython when done too. Sounds like a project to keep an eye on! > Now the question is will this make Vista run faster? Nothing could do that ;-) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Interfacing python and C
MRAB wrote: > steve William wrote: > > Hi All, > > > > I'm using SWIG for the first time and I am facing some problems with > > user defined header files. I'm trying to use my own header file in a C > > program which would be interfaced with python. > > > > The header file is test.h: > > /#include > > > > int fact(int n) { > > if (n <= 1) return 1; > > else return n*fact(n-1); > > }/ > > > > The C program is test1.c: > > /#include > > #include "test.h" > > > > int calc_fact(int a) > > { > > return (fact(a)); > > }/ > > > > The interface file is test1.i: > > /%module test1 > > > > %{ > > #include "stdio.h" > > #include "test.h" > > %} > > > > int calc_fact(int a);/ > > > > The commands that I used to generate the wrappers are: > > /swig -python test1.i > > gcc -c test1.c test1_wrap.c -I/usr/include/python2.5 > > -I/usr/lib/python2.5/config > > g++ -shared test1_wrap.o -o _test1.so/ gcc not g++ here link test1.o also gcc -shared test1.o test1_wrap.o -o test1.so > > When I try to import test1, I get an error saying: > > />>> import test1 > > Traceback (most recent call last): > > File "", line 1, in > > File "test1.py", line 22, in > > import _test1 > > ImportError: ./_test1.so: undefined symbol: calc_fact/ Try nm or objdump on the .so to see what symbols are in it. > > I'm not sure why this is happening because when I try without user > > defined header file, it works. Also the error says that it does not > > recognize /calc_fact/ which is a function that I want to access from > > python and is declared in the interface file. > > > > Is there any specific way in which user defined headers need to be > > declared in the interface file? Should the user defined header be placed > > in the /usr/include directory? > > > > Any help on this is highly appreciated. My advice to you is to compile the C stuff into a .so and use ctypes instead of swig. You then write the interface code in python not C and you'll have a lot more fun! cython is very useful in this area too provided you don't mind an extra dependency. If you are developing C code from scratch to use with python, then write it in cython instead! > Should you be putting a function body in a header file? No -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to port Python to an ARM running NucleusPlus OS
Thomas Ng wrote: > I'm planning to port the Python to run on an embedded device running > Nucleus+ OS on ARM7 processor. > > Can anyone help to advise how to get started? > > Where I am now: I've downloaded the Python 2.6 from SVN and able to > compile it using VC++. But no idea how to proceed from here. Since I > am not familiar with Linux, I would prefer to do the porting work on > Windows. Well... Nucleus isn't Linux, it is a RTOS. If it has enough POSIX interfaces you probably can port python but I wouldn't have thought it would be easy. Try to compile python in the cross compiling environment and see what happens! However if you are running Nucleus with Linux and want to run python in the Linux bit of it then I'd suggest to use the packages available for the Linux side of it. (Eg if it is running debian then apt-get install python). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: C extension using GSL
Gabriel Genellina wrote: > En Fri, 27 Mar 2009 03:10:06 -0300, jesse escribió: > > > I give up. I cannot find my memory leak! I'm hoping that someone out > > there has come across something similar. Let me lay out the basic > > setup: > > [...] > > 4) C: A PyList object, L, is created (new reference!). This will hold > > the solution vector for the ODE > > [...] > > 7) C: Return L to Python with return Py_BuildValue("N", L). > > I don't know the "N" format, but if Py_BuildValue returns a new reference, > you're leaking a reference to the L list. > Why don't you return L directly? > > You can use sys.getrefcount in Python to see the reference count for an > object: > > py> from sys import getrefcount as rc > py> x = object() > py> rc(x) > 2 # the name x, and a temporary reference as parameter > py> rc([]) > 1 # only the temporary reference > py> x = y = [] > py> rc(x) > 3 > py> x = () > py> rc(x) > 954 # the empty tuple is shared That reminds me, you can use the gc module to show all your objects that are in use, which can help with memory leaks. eg something like http://code.activestate.com/recipes/457665/ -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Code anntotations (copyright, autor, etc) in your code
mattia wrote: > Hi all, which are the usual comments that you put at the beginning of > your code to explain e.g. the author, the usage, the license etc? > > I've found useful someting like: > > #- > # Name:About.py > # Purpose: > # > # Author: > # > # Created: 2009 > # Copyright: (c) 2009 > # Licence: GPL > > #- > > others put something like > > __author__ = "Name Surname" > __year__ = 2009 > > What do you use? __version__ is suggested in PEP 8 http://www.python.org/dev/peps/pep-0008/ __author__, __date__ and __credits__ are both understood by pydoc but I haven't actually seen that in a document only in the source code! I think the others are just conventions and are not actually used by anything, but I'd be interested to be proved wrong! I tend to use __author__ = "Nick Craig-Wood " __version__ = "$Revision: 5034 $" __date__ = "$Date: 2009-02-03 16:50:01 + (Tue, 03 Feb 2009) $" __copyright__ = "Copyright (c) 2008 Nick Craig-Wood" With __version__ and __date__ being version control tags (in this case svn) Pydoc produces this from the above ---- [snip] DATA __author__ = 'Nick Craig-Wood ' __copyright__ = 'Copyright (c) 2008 Nick Craig-Wood' __date__ = '$Date: 2009-02-03 16:50:01 + (Tue, 03 Feb 2009) $' __version__ = '$Revision: 5034 $' VERSION 5034 DATE $Date: 2009-02-03 16:50:01 + (Tue, 03 Feb 2009) $ AUTHOR Nick Craig-Wood pydoc pydoc shows an example of what __credits__ looks like -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: c.l.py dead, news at 11 (was Re: Mangle function name with decorator?)
Aahz wrote: > Well, yes, but that's simply the nature of online fora (I originally > wrote "nature of Usenet", but I think it's more general than that). From > my POV, if you're going to call it a "decline", you need to provide more > evidence than some people leaving and others arriving. I think that the > sheer volume and quality of posts to c.l.py is evidence that c.l.py is > not declining. c.l.py is my favourite usenet group and has been for some time. I've been doing usenet for 16 years now! I enjoy reading about problems and problems solved. I enjoy helping people where I can and especially learning new things when helping people - I think it has contributed enormously to my development as a python programmer. When I ask a question of c.l.py I find the answers to be informative and enlightening. Even after quite a few years of python programing I'm still learning new things from c.l.py As a long time usenet user I find it easy to ignore the occasional flame wars. Posters with the wrong sort of attitude are brought gently into line by the majority. If usenet groups had ratings I'd give c.l.py 5 stars. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: C extension using GSL
jesse wrote: > I give up. I cannot find my memory leak! I'm hoping that someone out > there has come across something similar. Let me lay out the basic > setup: > > I'm performing multiple simulations on a model. Each iteration > involves solving a system of differential equations. For this I use > the GNU Scientific Library (GSL) -- specifically the rk4imp ODE > solver. After the ODE is solved the array is returned to python and is > analyzed. As you may have guessed, my problem is that over the course > of the iterations the memory keeps climbing until python crashes. > > Note: *The extension does not keep running.* It returns object (a > list) and is done until the next iteration. AFAIK, any memory > allocated during execution of the extension should be released. > Question: Since the extension was run from within python is memory > allocated within an extension part of python's heap? No, on the normal C heap > Would this have > an adverse or unpredictable affect on any memory allocated during the > running of the extension? If the library passes you data and ownership of a heap block, you'll need to free it. > One hypothesis I have, since most other > possibilities have been eliminated, is that GSL's creation of it's own > data structures (eg., gsl_vector) is messing up Python's control of > the heap. Is this possible? Only if GSL is buggy > If so, how would one be able to fix this > issue? Valgrind > It may help some nice folks out there who are good enough to look at > this post if I layout the flow, broken up by where stuff happens > (i.e., in the Python code or C code): > > 1) Python: Set up the simulation and basic data structures (numpy > arrays with initial conditions, coupling matrices for the ODE's, dicts > with parameters, etc). > 2) Python: Pass these to the C extension > 3) C: Python objects passed in are converted to C arrays, floats, > etc. > 4) C: A PyList object, L, is created (new reference!). This will hold > the solution vector for the ODE > 5) C: Initialize GSL ODE solver and stepper functions. Solve the ODE, > at each step use PyList_Append(L, current_state) to append the current > state to L. > 6) C: After the ODE solver finishes, free GSL objects, free coupling > matrices, free last state vector. > 7) C: Return L to Python with return Py_BuildValue("N", L). > 8) Python: Convert returned list L to array A, delete L, work with A. > 8.1) Python: Step 8) includes plotting. (I have a small suspicion that > matplotlib holds onto lots of data, but I've used clf() and close(fig) > on all plots, so I think I'm safe here. ) > 8.2) Python: save analysis results from A, save A. (At this point > there should be no more use of A. In fact, at point 8) in the next > iteration A is replaced by a new array.) > 9) Python: Change any parameters or initial conditions and goto 1). At every point a memory allocation is made check who owns the memory and that it is freed through all possible error paths. Valgrind will help you find the memory leaks. It works well once you've jumped the flaming hoops of fire that setting it up is! Another thing you can try is run your process untill it leaks loads, then make it dump core. Examine the core dump with a hex editor and see what it is full of! This technique works suprisingly often. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Bash-like brace expansion
Peter Waller wrote: > Okay, I got fed up with there not being any (obvious) good examples of > how to do bash-like brace expansion in Python, so I wrote it myself. > Here it is for all to enjoy! Interesting! Might I suggest some unit tests? You can then test all the corner cases (unmatched brackets, empty brackets, etc) and be sure it works exactly as specified. doctest is cool for this kind of stuff. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: udp package header
mete wrote: > I got a problem. İ want to send udp package and get this package (server and > clinet ). it's easy to python but i want to look the udp header how can i > do ? There is pretty much nothing in a UDP packet header except from_address, to_address, from_port and to_port. You should already know to_address and to_port and socket.recv will give you the from_address and from_port socket.recvfrom(bufsize[, flags]) Receive data from the socket. The return value is a pair (string, address) where string is a string representing the data received and address is the address of the socket sending the data. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Using python 3 for scripting?
Alan G Isaac wrote: > On 3/22/2009 12:41 PM Chris Rebert apparently wrote: > > 2.6.1, the latest non-3.x release is probably best. Most libraries > > haven't been ported to 3.x yet, so Python 3 has yet to become > > widespread. > > This seems slightly optimistic to me. Until a week ago, there was > not a NumPy release for 2.6. There is still not a SciPy release > for 2.6. Most dismaying, the SimpleParse guys need a little help > compiling SimpleParse for 2.6 and have not yet gotten that help. > (Is anyone listening?) > > So 2.5.4 is still perhaps the safest bet, even though it is more > awkward for writing code close to Python 3 syntax. I tend to target whatever is in Debian stable, which starting from this month is 2.5 (recently upgraded from 2.4). 2.6 or 3.x is nowhere to be seen in Debian stable, testing or unstable :-( -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone wrote: > On Mon, 23 Mar 2009 05:30:04 -0500, Nick Craig-Wood > wrote: > >Jean-Paul Calderone wrote: > > [snip] > >> > >> In the case of a TCP to serial forwarder, you don't actually have to > >> implement either a producer or a consumer, since both the TCP connection > >> and the serial connection are already both producers and consumers. All > >> you need to do is hook them up to each other so that when the send buffer > >> of one fills up, the other one gets paused, and when the buffer is empty > >> again, it gets resumed. > > > >I eventually came up with this which seems to work, but I'm not sure > >it is the best way of doing it as it had to mess about with the > >twisted internals to get the number of bytes in the serial port > >output buffer. > > This is sort of on the right track. Here's how to do it without > touching implementation details: Thank you for that! See below for the complete prog with your suggested modifications. That seems to solve the problem - I can see the writer pausing and unpausing at the serial port rate. write 16446 write 6430 pause producing resume producing write 65536 write 56724 pause producing resume producing write 65536 write 65536 pause producing It has exposed a problem with the sender not throttling properly now, but I guess that is progress! Thanks for your help Here is the complete program FYI with your suggested mods. #!/usr/bin/python """Transfer data between a serial port and one (or more) TCP connections. options: -h, --help:this help -p, --port=PORT: port, a number, default = 0 or a device name -b, --baud=BAUD: baudrate, default 115200 -t, --tcp=PORT: TCP port number, default 1234 -l, --log: log data streams to 'snifter-0', 'snifter-1' -L, --log_name=NAME: log data streams to '-0', '-1' """ import sys import getopt from twisted.internet import reactor, protocol, serialport from zope.interface import implements from twisted.internet import protocol, interfaces # FIXME set serial buffer size? SEND_LIMIT class SerialPort(protocol.Protocol): """Create a serial port connection and pass data from it to a known list of TCP ports.""" def __init__(self, port, reactor, baudrate, log_name=None): self.tcp_ports = [] self.serial = serialport.SerialPort(self, reactor, port, baudrate, rtscts=0) self.serial.registerProducer(self, True) self.paused = False self.log = None if log_name is not None: self.log = file('%s-0' % log_name, 'w') def add_tcp(self, tcp_port): """Add a TCPPort to those receiving serial data.""" if self.paused: tcp_port.transport.pauseProducing() self.tcp_ports.append(tcp_port) def del_tcp(self, tcp_port): """Remove a TCPPort from the those receiving serial data.""" self.tcp_ports.remove(tcp_port) def write(self, data): """Write data to the serial port.""" self.serial.write(data) if self.log: self.log.write(data) def pauseProducing(self): """Pause producing event""" print "pause producing" self.paused = True for port in self.tcp_ports: port.transport.pauseProducing() def resumeProducing(self): """Resume producing event""" print "resume producing" self.paused = False for port in self.tcp_ports: port.transport.resumeProducing() def stopProducing(self): """Stop producing event""" print "stop producing" def dataReceived(self, data): """Pass any received data to the list of TCPPorts.""" for tcp_port in self.tcp_ports: tcp_port.write(data) class TCPPort(protocol.Protocol): """Create a TCP server connection and pass data from it to the serial port.""" def __init__(self, serial, log_name, index): """Add this TCPPort to the SerialPort.""" self.serial = serial self.serial.add_tcp(self) self.log = None if log_name is not None: self.log = file('%s-%d' % (log_name, index+1), 'w') def __del__(self): """Remove this TCPPort from the SerialPort.""" self.serial.del_tcp(self) def dataReceived(self, data): """Pass received data
Re: Async serial communication/threads sharing data
Jean-Paul Calderone wrote: > On Sun, 22 Mar 2009 12:30:04 -0500, Nick Craig-Wood > wrote: > >I wrote a serial port to TCP proxy (with logging) with twisted. The > >problem I had was that twisted serial ports didn't seem to have any > >back pressure. By that I mean I could pump data into a 9600 baud > >serial port at 10 Mbit/s. Twisted would then buffer the data for me > >using 10s or 100s or Megabytes of RAM. No data would be lost, but > >there would be hours of latency and my program would use up all my RAM > >and explode. > > > >What I wanted to happen was for twisted to stop taking the data when > >the serial port buffer was full and to only take the data at 9600 > >baud. > > This is what Twisted's producers and consumers let you do. There's a > document covering these features: > > http://twistedmatrix.com/projects/core/documentation/howto/producers.html > > In the case of a TCP to serial forwarder, you don't actually have to > implement either a producer or a consumer, since both the TCP connection > and the serial connection are already both producers and consumers. All > you need to do is hook them up to each other so that when the send buffer > of one fills up, the other one gets paused, and when the buffer is empty > again, it gets resumed. I eventually came up with this which seems to work, but I'm not sure it is the best way of doing it as it had to mess about with the twisted internals to get the number of bytes in the serial port output buffer. class SerialPort(protocol.Protocol): """Create a serial port connection and pass data from it to a known list of TCP ports.""" def __init__(self, port, reactor, baudrate, log_name=None): self.tcp_ports = [] self.serial = serialport.SerialPort(self, reactor, port, baudrate, rtscts=0) self.log = None if log_name is not None: self.log = file('%s-0' % log_name, 'w') def write(self, data): """Write data to the serial port.""" self.serial.write(data) if self.log: self.log.write(data) self.throttle() def throttle(self): """ Pause the inputs if there is too much data in the output buffer """ bytes_in_buffer = len(self.serial.dataBuffer) + self.serial._tempDataLen too_full = bytes_in_buffer > 1024 for tcp_port in self.tcp_ports: if too_full: tcp_port.transport.pauseProducing() else: tcp_port.transport.resumeProducing() if too_full: reactor.callLater(0.1, self.throttle) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone wrote: > It's true that the serial port support in Twisted isn't the most used > feature. :) These days, serial ports are on the way out, I think. That > said, much of the way a serial port is used in Twisted is the same as the > way a TCP connection is used. This means that the Twisted documentation > for TCP connections is largely applicable to using serial ports. The only > major difference is how you set up the "connection". You can see examples > of using the serial port support here (one of them seems to suggest that > it won't work on Windows, but I think this is a mistake): > >http://twistedmatrix.com/projects/core/documentation/examples/gpsfix.py >http://twistedmatrix.com/projects/core/documentation/examples/mouse.py > > The 2nd to last line of each example shows how to "connect" to the serial > port. These basic documents describe how to implement a protocol. Though > in the context of TCP, the protocol implementation ideas apply to working > with serial ports as well: > >http://twistedmatrix.com/projects/core/documentation/howto/servers.html >http://twistedmatrix.com/projects/core/documentation/howto/clients.html > > You can ignore the parts about factories, since they're not used with serial > ports. I wrote a serial port to TCP proxy (with logging) with twisted. The problem I had was that twisted serial ports didn't seem to have any back pressure. By that I mean I could pump data into a 9600 baud serial port at 10 Mbit/s. Twisted would then buffer the data for me using 10s or 100s or Megabytes of RAM. No data would be lost, but there would be hours of latency and my program would use up all my RAM and explode. What I wanted to happen was for twisted to stop taking the data when the serial port buffer was full and to only take the data at 9600 baud. I never did solve that problem :-( -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: script files with python (instead of tcsh/bash)?
Esmail wrote: > thanks for including the script, that really helps. Nice way of > finding files. Python has lots of useful stuff like that! > Two quick questions: > > As a replacement for grep I would use the re module and its > methods? The re module works on strings not files, but basically yes. Note that the re module uses a superset of the "grep -E" regexps, and they are almost identical to those used in perl and php. Here is a simple grep in python #!/usr/bin/python import re import sys import fileinput pattern = sys.argv.pop(1) for line in fileinput.input(): if re.search(pattern, line): print line.rstrip() Save in a file called grep.py then you can do $ ./grep.py dizzy *.py self.dizzy = 0 if self.dizzy: $ ls | ./grep.py '\d{2}' linux-image-2.6.24-21-eeepc_2.6.24-21.39eeepc1_i386.deb linux-ubuntu-modules-2.6.24-21-eeepc_2.6.24-21.30eeepc6_i386.deb > What about awk which I regularly use to extract fields based on position > but not column number, what should I be using in Python to do the > same? I presume you mean something like this ... | awk '{print $2}' In python, assuming you've got the line in the "line" variable, then In python an equivalent of the above would be import fileinput for line in fileinput.input(): print line.split()[1] Note that the fileinput module is really useful for making shell command replacements! > The other things I need to do consist of moving files, manipulating file > names and piping outputs of one command to the next, so I'm digging into > the documentation as much as I can. Read up on the os module and the subprocess module. You'll find you need to do much less piping with python as with shell because it has almost everything you'll need built in. Using built in functions is much quicker than fork()-ing an external command too. > So much to learn, so little time (but so much fun!) ;-) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: 3.0 - bsddb removed
Sean wrote: > Anyone got any thoughts about what to use as a replacement. I need > something (like bsddb) which uses dictionary syntax to read and write an > underlying (fast!) btree or similar. sqlite. bsddb gave me no end of trouble with threads, but sqlite worked brilliantly. You would need to make a dictionary interface to sqlite, eg http://code.activestate.com/recipes/576638/ Or do something a bit simpler yourself. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: script files with python (instead of tcsh/bash)?
Esmail wrote: > I am wondering if anyone is using python to write script files? Yes! > Right now I have a bigg'ish bash/tcsh script that contain some grep/awk > command plus various files are processed and created, renamed and > moved to specific directories. I also write out some gnuplot scripts > that later get executed to generate .jpg images. Almost any script that contains a loop I convert into python. > In any case, the scripts are starting to look pretty hairy and I was > wondering if it would make sense to re-write them in Python. I am not > sure how suitable it would be for this. With python you get the advantages of a language with a very clear philosophy which is exceptionally easy to read and write. Add a few classes and unit tests to your scripts and they will be better than you can ever achieve with bash (IMHO). > I've looked around the web w/o much luck for some examples but come > short. Any comments/suggestions? Here is a short script I wrote to recover my slrn spool when my disk fills up. I could have written it in bash but I think it is much clearer as python, and doesn't shell out to any external commands. It searches through the spool directory looking for empty .minmax files. It then finds the lowest and highest numeric file name (all the files are numeric) and writes the minmax file with that. #!/usr/bin/python """ rebuild all the .minmax files for slrnpull after the disk gets full """ import os spool = "/var/spool/slrnpull" def main(): for dirpath, dirnames, filenames in os.walk(spool): if ".minmax" not in filenames: continue minmax_path = os.path.join(dirpath, ".minmax") if os.path.getsize(minmax_path) != 0: print "Skipping non empty %r" % minmax_path continue print dirpath digits = [ int(f) for f in filenames if f.isdigit() ] if not digits: digits = [0] digits.sort() start = digits[0] end = digits[-1] f = open(minmax_path, "w") f.write("%s %s" % (start, end)) f.close() print "done" if __name__ == "__main__": main() -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: cross compile Python to Linux-ARM
jefm wrote: > We are looking to use Python on an embedded Linux ARM system. > What I gather from googling the subject is that it is not that > straight forward (a fair amount of patching & hacking). > Nobody out there that has done it claims it is easy, which makes me > worried. > > I haven't seen a description on porting Python 2.6 or 3.0 yet. Is it > much different than for the earlier versions (the latest I have seem > is Python 2.5). > > Does it matter whether Python is cross compiled to Linux 2.4 or Linux > 2.6 ? > > Can anyone point to a howto they know works well ? I gave up trying to cross compile and just used the python from debian ARM which works very well. > What are the chances of an 'officially' supported ARM-Linux Python > distribution ? Depending on your definition of "officially" and "supported" :- http://packages.debian.org/lenny/arm/python2.5/download > (or is it safer to wait for industrial spec Intel Atom boards to avoid > the cross compilation altogether ? We often compile stuff on our 200 MHz TS-7200 boards. We NFS mount the build environment, type make then go out for lunch ;-) For builds of our application (which embeds python) we normally use the headers and libraries from the debian ARM packages and cross compile on something a big beefier. > How is the performance and stability of a working Python on an > embedded ARM-Linux system ? Works very well. > Does cross compiling Python automatically include the standard > Python library, or is that yet another adventure ? If you use the debian compiled version then you get the lot. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Python to Perl transalators
Armin wrote: > On Wednesday 18 March 2009 11:01:00 Boris Borcic wrote: > > Armin wrote: > > >> > > >> Why on earth would you want to? That'd be like translating Shakespeare > > >> into a bad rap song! > > >> > > > > > > lol, actually I would prefer a rap song over Shakespeare, so your analogy > > > doesn't work there ;) > > > > Why, some people do prefer Perl over Python, so what's wrong with the > > analogy ? > > > > That would mostly apply to people who haven't coded in Python for one reason > or another. Perhaps the OP is looking for something like this http://pleac.sourceforge.net/pleac_python/index.html Which is a sort of Rosetta stone for perl and python ;-) (The perl cookbook translated into python.) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: packaging
> andrew thanks andrew, good advice, I should probably use that throughout our code. btw, hope the world is treating you well, long time no see... -craig -- http://mail.python.org/mailman/listinfo/python-list
packaging
we have software we are putting into package form. So far, all the code was in local py files and we imported between the modules as you'd think. Now with the package ("ourpackage") we are addressing how import affects the importing module. if "ourpackage" __init__.py itself does regular imports of the key modules, like "ourmodule" (containing class OurClass)... it's a bit of a pain for the user. one imports ourpackage, and then need to get OurClass from ourmodule still, i.e.: import ourpackage inst = ourpackage.ourmodule.OurClass() Instead, I think we want "import package" to preserve the sort of namespace our loose python files provided, so: import ourpackage inst = ourpackage.OurClass() I think the way to do this, and it seems a legit use of a somewhat dangerous form of import, to in ourpackage's __init__.py do this: from ourmodule import * So that the second form works. If anyone has a comment on that I'm interested, either that it won't work, or to contradict my idea that a wildcarded import is appropriate in this place as we are trying to fill a flattened namespace. But the reason I'm asking is that it's also been suggested that we should include everything in a single module, so, ourpython1.py and ourpython2.py, etc, all get moved to ourpython.py. I very much dislike that idea for various (probably obvious) reasons. On the other hand I do want to adopt whatever are de facto standards for this sort of thing (especially from the user pov but also from the developer's)... so any comment are appreciated. I've been using python for a few years now but this is the first time we are forming it in the shape of a proper package. cheers and thanks. -craig -- http://mail.python.org/mailman/listinfo/python-list
Re: Special keyword argument lambda syntax
Beni Cherniavsky wrote: > This proposal outrageously suggests a special syntax for in-line > functions passed as keyword arguments:: > > >>> sorted(range(9), key(n)=n%3) > [0, 3, 6, 1, 4, 7, 2, 5, 8] > > The claim is that such specialization hits a syntax sweet spot, and > that this use case is frequent enough to contemplate eventually making > it the only in-line function syntax. -1 from me. I think that lambda / inline functions should be discouraged as it moves python away from, "there should be one-- and preferably only one --obvious way to do it." IMHO Guido was right in his original impulse to kill this second class way of making functions... I would write the above as def compute_key(n): "Compute the sort key so that x, y and z are true" return n % 3 sorted(range(9), key=compute_key) Which I think is clearer and more obvious. It gives you the opportunity for a docstring also. Yes it is a bit more typing, but who wants to play "code golf" all day? -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Raw String Question
Jim Garrison wrote: > >>> r"a\b" >'a\\b' > >>> r"a\" >SyntaxError: EOL while scanning string literal (, line 1) > >>> r"a\ " >'a\\ ' > >>> r"a\"" >'a\\"' > > It seems the parser is interpreting the backslash as an escape > character in a raw string if the backslash is the last character. > Is this expected? Yes http://docs.python.org/reference/lexical_analysis.html#string-literals Specifically, a raw string cannot end in a single backslash (since the backslash would escape the following quote character). The usual way round this is like this >>> r"a" "\\" 'a\\' >>> Which isn't terribly elegant, but it doesn't happen very often. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is lambda allowed as a key in a dict?
On Mar 10, 1:39 pm, Paul Rubin <http://phr...@nospam.invalid> wrote: > Craig Allen writes: > > it raises an interesting question about why doesn't it. I can think > > of practical answers to that, obviously, but in principle, if a > > function compiles to exactly the same byte code, you obviously do not > > need two copies of it, and like strings shouldn't an identical > > function have the same id? > > Identical strings don't necessarily have the same id: > > >>> a = "a"*1000 > >>> b = "a"*1000 > >>> id(a),id(b) > (137143648, 137144680) > >>> a==b > True interesting, I thought they were supposed to. -- http://mail.python.org/mailman/listinfo/python-list
Re: Ban Xah Lee
> There you go: a 30-second psychological diagnosis by an > electrical engineer based entirely on Usenet postings. Â It > doesn't get much more worthless than that... > > -- > Grant rolf but interesting post nonetheless. I have been really somewhat fascinated by AS since I heard of it about a decade ago. There are many among us, with interesting ideas, occasionally savant level insight into certain abstractions, which often they can not communicate but which lie there for those that can communicate or come to understand nonetheless. having said that, none of this forgives rudeness or implies people have to tolarate it due to a person's condition, or even due to trying to help them achieve their potential (and thus get something productive out of it ourselves as well)... that is, if you have these communications problems you have to realize it, thank god you are functional, and just that alone will help you communicate. me, also IANAP, also working from usenet and an asperger's book I read (and google)... -- http://mail.python.org/mailman/listinfo/python-list
Re: Is python worth learning as a second language?
On Mar 9, 12:43 am, ZikO wrote: > Hi > > I hope I won't sound trivial with asking my question. > > I am a C++ programmer and I am thinking of learning something else > because I know second language might be very helpful somehow. I have > heard a few positive things about Python but I have never writen any > single line in python so I do not know this language at all. > > Do you think python would be good complementary language for C++? Do you > think it's worth learning it or let's say try Java? and how difficult it > would be for me if I know C++ pretty well I would say? > > Thanks I'm not even going to read the replies first because I have my own. I was a C and C++ programmer exclusively for over a decade. During that time I had a "whatever tool for the problem" approach to language selection but C++ continued to be the best for the sort of thing I was doing. During that time I continued to learn other languages at least enough to consider them. I appreciated the role of interpreted languages were filling, but also never felt comfortable in them. Python, imo, is an excellent language to learn as a C++ programmer. It is relatively easy to extend with C/C++ and so works together well. When you find yourself writing some configuration language, you'll be able to use python instead. Also, I have found that Python helped open my mind a bit about object orientation and to realize that while the compile-time decisions in C++ are great for the C/C++ linking model, and provides a certain sort of power and control, that it also really does (as many had complained to me) take a few blows at how you really want OO to work. So I love python's OO and things which can be done structurally in C++ (like metaclass programming) but with which the C++ syntax is not cooperative, and which is very much harder to safely extend modularity too (I could go into detail but why bother here, it's either already clear what I mean or isn't that important)... Finally, Python is a multiparadigmed language, like C++. It very much seems to believe in Bjarne's original trust the programmer outlook on languages. As a C++ programmer I enjoyed the responsibility and power of choosing one's paradigms at the outset of a project. Such choices are best made consciously, one learns a lot about their project and develops a lot of philosophical standards for the design and implementation by having to think first "what models and programming paradigms will we adopt". It makes you think what sort of code you will be writing often, and which idioms will be efficient and maintainable. Honestly, I've become more of a Python fan than I am really comfortable with... it can't be as good as I think. -craig -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is lambda allowed as a key in a dict?
> I think the point is that function objects compare by object identity, > so the two lambdas you use above are not equal even though they have the > same code. it raises an interesting question about why doesn't it. I can think of practical answers to that, obviously, but in principle, if a function compiles to exactly the same byte code, you obviously do not need two copies of it, and like strings shouldn't an identical function have the same id? This is merely philosophical, I don't see the value in making this so (a slight optimizaton and perhaps better conformity to other python features), but I do think it's an interesting question. cheers. -- http://mail.python.org/mailman/listinfo/python-list
Re: A better way to timeout a class method?
John O'Hagan wrote: > Is there a concise Pythonic way to write a method with a timeout? > > I did this: > > class Eg(object): > > def get_value(self, timeout): > > from threading import Timer > self.flag = True > > def flag_off(): > self.flag = False > timer = Timer(timeout, flag_off) > timer.start() > > while self.flag: > #do stuff for a long time to get some value > if condition: #if we ever get the value > self.value = value > break > > but it seems hackish to have a special "flag" attribute just to manage the > timeout within the the method. How about something like this from threading import Timer class Duration(object): def __init__(self, duration): self.running = True self.timer = Timer(duration, self.set_finished) self.timer.setDaemon(True) self.timer.start() def set_finished(self): self.running = False def __nonzero__(self): return self.running from time import sleep def long_function(): duration = Duration(5) i = 0 print "starting" while duration: print i i += 1 sleep(1) print "finished" long_function() Which prints starting 0 1 2 3 4 finished -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: A better way to timeout a class method?
Marco Mariani wrote: > John O'Hagan wrote: > > > Is there a concise Pythonic way to write a method with a timeout? > > No need for threading. Just define a signal handler and call signal.alarm(). > > See the example at the end of the page: > > http://docs.python.org/library/signal.html Won't work on windows and there is only one sigalarm timer, so you can't nest them :-( -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Any c header parser for generate ctypes module?
Diez B. Roggisch wrote: > Victor Lin wrote: > > I am writing python binding for some c library. It is really a super > > boring job. Copy... paste... modify.. copy paste...modify I am > > wondering, I am a programmer, why I can't do this job like a > > programmer? So I think the best way to write binding for those c > > libraries, is to write a parser that parse header of c library and > > generate ctypes python module automatically. My question is, is there > > any available tools can achieve this? If not, what tool can I use to > > such a job easily. I need a c parser, is there any C parser written in > > python? > > GCCXML is usually used to create ctypes-structures from headers. Look at http://pypi.python.org/pypi/ctypeslib/ And the h2xml and xml2py scripts that are part of it. You'll need gccxml too as Diez pointed out. A definite time saver! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Is python worth learning as a second language?
ZikO wrote: > I am a C++ programmer and I am thinking of learning something else > because I know second language might be very helpful somehow. I have > heard a few positive things about Python but I have never writen any > single line in python so I do not know this language at all. > > Do you think python would be good complementary language for C++? Yes > Do you think it's worth learning it or let's say try Java? Jave is basically C++ with the hard bits taken out. I wouldn't bother. > and how difficult it would be for me if I know C++ pretty well I > would say? Go here http://www.diveintopython.org/ Download the PDF or buy the book. You'll be writing your first python program in about 5 minutes! We use lots of python embedded into our humungous C++ app. Gradually the dev team has moved from writing C++ to writing python as much of the time as possible and C++ only when necessary. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Opening for Python Programmer at Newport Beach
On Mar 3, 10:17 am, Cool Dude wrote: > Hello , > This is Aniket from Techclique, a New Jersey based software > development and IT consulting firm providing top quality technical and > software professionals on a permanent and contractual basis to > Government and commercial customer including fortune 500 companies and > most of states and federal governments. I am contacting you to see if > you are comfortable to the skills mentioned below, please forward > resume to this e-mail address ani...@techclique.com > > Python Programmer (someone with industry knowledge/asset mgt) > Location: Newport Beach, CA (Need Local) > Duration: 6+ months > Note: Communication MUST be perfect!, Plus submit with 3 professional > references > > Python programming skills (with web development and dynamically > generated charts/plots in particular) and working knowledge of Linux/ > UNIX Shell Scripts and SQL. > Knowledge of Python integration with C/C++ - a definite plus. > > Qualifications/Requirements > * 3+ years of Python programming on Linux/Unix platform; recent (2007) > required need programmer to write bot that doesn't multipost... please submit three references so I can check to see if you've ever multiposted on usenet. > * Programming skills building forms, lay-outs, charts, and graphing > required > * Designing, coding, and testing web based applications preferred. > * Strong organizational, oral and written communications skills > * High energy/self starter with the ability to work independently > within the firm*s demanding and highly focused environment > > Thanks & Regards, > Aniket > Techclique Inc. > Jersey City, NJ > Email: ani...@techclique.com > Yahoo IM : aniket_mitta...@yahoo.com > Contact No : 732-357-3844 -- http://mail.python.org/mailman/listinfo/python-list
Re: Using xreadlines
bearophileh...@lycos.com wrote: > Brett Hedges: > > My question is how do I go to a previous line in the file? xreadlines has a > > file.next() statement that gives the next line, and I need a statement that > > gives me the previous line.< > > In modern versions of Python you usually don't need xreadlines, > because files are iterable. > > If your files are small, you can just read all the lines in a list > with open(...).readlines(), and then just use the item of the list > with the n-1 index. There is also the linecache module http://docs.python.org/library/linecache.html Which may be useful... -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Extending Python Questions .....
Ben wrote: > On Feb 24, 11:31?am, Nick Craig-Wood wrote: > > So do you want to embed python into your code? > > > > I'm still not clear what you are trying to achieve with python, though > > I have a better idea what SLAG is now! > > Actually no, I want to EXTEND python using the lower levels of S-lang > screen modules. Ah! > My Modules are written in C and are a frame work for building pull- > down menus and data entry screens. > > Very nice for writing business applications. Think along the lines > of FoxPro and/or the "Screen" section in Cobol and you have a > pretty good idea of what i have done. You've got several choices. 1) Write a python module in C which interfaces with your C code. This is the traditional way, but it is quite time consuming and easy to slip up with reference counting. You need to learn the python API to use this. Will require compiling by your users (not a big hurdle if you make a setup.py). 2) Write an interface with ctypes If your C code presents itself as a shared object (dll/so) then this is a really nice way of using it. You don't have to write any C code or worry about any reference counting - you do it all in python. You'll need to learn how ctypes maps python onto C, but as you know C well you won't find this difficult. ctypes is included with python now. 3) Use Cython Cython is a sort of hybrid C and python which can be used for interfacing with C code. This requires learning exactly what you can and can't do in Cython (it is almost python but not quite). This will require your users to install Cython and compile stuff. Again easy if you make a setup.py but more stuff to install. 4) Use Swig A solid approach, quicker than 1) but not as quick as 2) or 3) This will require your users to install swig and a compiler. ... My prefered approach is 2) ctypes at the moment. I have used all 4 of the above approaches in the past though! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Extending Python Questions .....
Ben wrote: > No, It uses the the S-lang for video, and input control. However, SLAG > is more of an abstract layer on top of that. > > It has a Structures that contains menus and screens (menumodule / > screenmodule). One LOADS them up with parameters. such as creating > a new menu is like: > > OpenMenu( Company name, SubSystem, this program name, mode, bottom > status display) - Create initial menu structure Addtomenu(Menu > Block Set name, DISPLAY line, ID, type of program, password ID ) - > add to / update MENU blocks. runMenu() - Displays the whole create > menu structure. > > The Menu structure is done in pull downs and scrollable blocks in a > TUI (text User Interface) and using the S-lang screen library is > fully mouseable. > > The screen module works mych the same way, but with the abiltity to > open and close and work within "Sub Screens". > > For those who do not know, S-lang is a interpreted language much > like Python. However, there is s direth of library modules. The > original S-lang started out as library of screen of keyboard > modules, but has been expanded > > My SLAG project does not care in reality WHICH or what language, it > is simply handling menu and screen control. So do you want to embed python into your code? I'm still not clear what you are trying to achieve with python, though I have a better idea what SLAG is now! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: intermediate python csv reader/writer question from a beginner
Learning Python wrote: > anything related to csv, I usually use VB within excel to manipulate > the data, nonetheless, i finally got the courage to take a dive into > python. i have viewed a lot of googled csv tutorials, but none of > them address everything i need. Nonetheless, I was wondering if > someone can help me manipulate the sample csv (sample.csv) I have > generated: > > ,, > someinfo,,, > somotherinfo,,, > SEQ,Names,Test1,Test2,Date,Time,, > 1,Adam,1,2,Monday,1:00 PM,, > 2,Bob,3,4,Monday,1:00 PM,, > 3,Charlie,5,6,Monday,1:00 PM,, > 4,Adam,7,8,Monday,2:00 PM,, > 5,Bob,9,10,Monday,2:00 PM,, > 6,Charlie,11,12,Monday,2:00 PM,, > 7,Adam,13,14,Tuesday,1:00 PM,, > 8,Bob,15,16,Tuesday,1:00 PM,, > 9,Charlie,17,18,Tuesday,1:00 PM,, > > into (newfile.csv): > > Adam-Test1,Adam-Test2,Bob-Test1,Bob-Test2,Charlie-Test1,Charlie- > Test2,Date,Time > 1,2,3,4,5,6,Monday,1:00 PM > 7,8,9,10,11,12,Monday,2:00 PM > 13,14,15,16,17,18,Tuesday,1:00 PM > > note: > 1. the true header doesn't start line 4 (if this is the case would i > have to use "split"?) > 2. if there were SEQ#10-12, or 13-15, it would still be Adam, Bob, > Charlie, but with different Test1/Test2/Date/Time I'm not really sure what you are trying to calculate, but this should give you some ideas... import csv from collections import defaultdict reader = csv.reader(open("sample.csv")) result = defaultdict(list) for row in reader: # ignore unless first row is numeric if not row or not row[0].isdigit(): continue n, name, a, b, day, time = row[:6] print "n=%r, name=%r, a=%r, b=%r, day=%r, time=%r" % (n, name, a, b, day, time) result[(day, time)].append(n) writer = csv.writer(open("newfile.csv", "w")) for key, values in result.iteritems(): day, time = key values = values + [day, time] writer.writerow(values) This prints n='1', name='Adam', a='1', b='2', day='Monday', time='1:00 PM' n='2', name='Bob', a='3', b='4', day='Monday', time='1:00 PM' n='3', name='Charlie', a='5', b='6', day='Monday', time='1:00 PM' n='4', name='Adam', a='7', b='8', day='Monday', time='2:00 PM' n='5', name='Bob', a='9', b='10', day='Monday', time='2:00 PM' n='6', name='Charlie', a='11', b='12', day='Monday', time='2:00 PM' n='7', name='Adam', a='13', b='14', day='Tuesday', time='1:00 PM' n='8', name='Bob', a='15', b='16', day='Tuesday', time='1:00 PM' n='9', name='Charlie', a='17', b='18', day='Tuesday', time='1:00 PM' And leaves newfile.csv with the contents 1,2,3,Monday,1:00 PM 7,8,9,Tuesday,1:00 PM 4,5,6,Monday,2:00 PM -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Extending Python Questions .....
Ben wrote: > In My S-Lag Project called, SLAG, I have some function keys that get > mapped back to S-lang internal functions. > > My SLAG project works pretty much like Python (as does the S-Lang). > You write a S-lang script > that "imports" your extension. module - and all this gets run by the > shell/interpreter. > > I allow function keys to be mapped back internal function(s) inside of > the controlling program. > > My question is which Python C api Do I need to this with ? Do you mean like this? http://code.google.com/p/python-slang Not sure how well maintained it is though. > Do I need to worry about my reference counting since the Python > Program is in essence calling a function in itself? I'm not sure I understand you here, but in general you don't need to worry about reference counting in Python - it all happens behind the scenes. If you are writing a python extension in C then you do need to worry about reference counting - a lot! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: can multi-core improve single funciton?
sturlamolden wrote: > On Feb 10, 8:38?am, Paul McGuire wrote: > > > Even worse than linear, the function is recursive, which as I > > understand it, is inherently a no-no when looking for code that is > > parallel-friendly. > > There is no way to parallelize Fibonacci numbers computed with linear > time complexity, as we must know fib(n-1) to compute fib(n). But if we > were to use Oyster's recursive version, which has geometric > complexity, one could parallelize with a 'forkjoin' scheme. > > Anyhow, this runs in amortized linear time and would be a lot faster: > > def fib(n): > while True: >try: > return fib.seq[n] >except AttributeError: > fib.seq = [0, 1, 1] >except IndexError: > fib.seq.append( fib.seq[-2] + >fib.seq[-1] ) Or something which runs in O(1)... from gmpy import mpf, mpz, digits from math import log, sqrt root5_float = sqrt(5) phi_float = (1 + root5_float) / 2 log_phi_float = log(phi_float) log_root5_float = log(root5_float) log2 = log(2) def fibonacci_noniterative(n): """ Work out n'th Fibonacci number in an noniterative fashion using Binet's formula """ # Work out magnitude of answer bits = int(((n * log_phi_float - log_root5_float) / log2)*1.1) if bits < 0: bits = 0 root5 = mpf(5, bits).sqrt() phi = (mpf(1, bits) + root5) / mpf(2, bits) f_n = phi**n / root5 f_n = mpz(f_n + mpf(1, bits) / mpf(2, bits)) return long(f_n) It runs in O(1) if the O we are talking about is large integer arithmetic. It actually runs in more like O(log(n)**1.5) time. It is a lot faster than the iterative approach too which runs in O(log(n)**2) for any n > ~40. Times to calculate F(n) n, iterative, noniterative 1, 0.03, 0.16 2, 0.03, 0.14 4, 0.03, 0.12 8, 0.03, 0.16 16, 0.04, 0.09 32, 0.07, 0.10 64, 0.14, 0.10 128, 0.32, 0.11 256, 0.72, 0.14 512, 0.000157, 0.19 1024, 0.000361, 0.31 2048, 0.000881, 0.66 4096, 0.002504, 0.000178 8192, 0.007640, 0.000521 16384, 0.025362, 0.001572 32768, 0.090633, 0.004701 65536, 0.342724, 0.014132 131072, 1.335723, 0.045194 262144, 5.273360, 0.111201 524288, 20.791205, 0.301488 1048576, 82.617938, 0.833644 Here is the rest of the program just in case anyone wants to play def fibonacci_iterative(n): """ Work out n'th Fibonacci number in an iterative fashion """ a, b = 0, 1 while n > 0: a, b = a + b, a n -= 1 return a if __name__ == "__main__": # warmup fibonacci_noniterative(100) print " n, iterative, noniterative" for e in range(30): i = 2 ** e t0 = time() f_iterative = fibonacci_iterative(i) t_iterative = time() - t0 t0 = time() f_noniterative = fibonacci_noniterative(i) t_noniterative = time() - t0 print "%10d, %10.6f, %10.6f" % (i, t_iterative, t_noniterative) if f_iterative != f_noniterative: print "Calculation error" print "iterative", f_iterative print "non iterative", f_noniterative print "difference", f_iterative-f_noniterative -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Threading and tkinter
> The statement > > x=x+1 > > (which, by the way, should stylistically be written > > x = x + 1 > yes I was wondering what "x=x+1" meant until you translated it... oh, "x = x + 1" of course! I thought to myself. Oh wait no I'm sarcastic. -- http://mail.python.org/mailman/listinfo/python-list
Re: How do I declare global vars or class vars in Python ?
Linuxguy123 wrote: > How do I do this in Python ? > > # > declare A,B > > function getA > return A > > function getB > return B > > function setA(value) > A = value > > function setB(value) > B = value > > main() > getA > getB > dosomething > setA(aValue) > setB(aValue) > > > The part I don't know to do is declare the variables, either as globals > or as vars in a class. How is this done in Python without setting them > to a value ? Variables can have any value in python so if you want to pre-declare then you set them to None normally. As a class :- class Stuff(object): def __init__(self): self.A = None self.B = None def getA(self): return self.A def getB(self): return self.B def setA(self, value): self.A = value def setB(self, value): self.B = value >>> a = Stuff() >>> print a.getA() None >>> print a.getB() None >>> # dosomething ... a.setA("aValue") >>> a.setB("aValue") >>> print a.getA() aValue >>> print a.getB() aValue >>> Note that in python we don't normally bother with getA/setA normally, just use self.A, eg class Stuff(object): def __init__(self): self.A = None self.B = None def main(self): print self.A print self.B # dosomething self.A = "aValue" self.B = "aValue" print self.A print self.B >>> a = Stuff() >>> a.main() None None aValue aValue >>> If you need (later) A to be a computed value then you turn it into a property, which would look like this. (Note the main method is identical to that above). class Stuff(object): def __init__(self): self._A = None self.B = None def _getA(self): print "Getting A" return self._A def _setA(self, value): print "Setting A" self._A = value A = property(_getA, _setA) def main(self): print self.A print self.B # dosomething self.A = "aValue" self.B = "aValue" print self.A print self.B >>> a = Stuff() >>> a.main() Getting A None None Setting A Getting A aValue aValue >>> -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to determine if a string is a number
pyt...@bdurham.com wrote: > Thanks for everyone's feedback. I believe my original post's code > (updated following my signature) was in line with this list's feedback. > > Christian: Thanks for reminding me about exponential formats. My updated > code accounts for these type of numbers. I don't need to handle inf or > nan values. My original code's except catches an explicit ValueError > exception per your concern about bare excepts. > > Malcolm > > > # str_to_num.py > > def isnumber( input ): > try: > num = float( input ) > return True > > except ValueError: > return False That is a fine solution. Last time I had to solve this I had a specific format of number to parse - I didn't want to include all the python formats. This is what I came up with... This particular code returns the converted number or 0.0 - adjust as you see fit! import re _float_pattern = re.compile(r"^\s*([-+]?(\d*\.)?\d+([eE][-+]?\d+)?)") def atof(value): """ Convert a string to an float in the same way the c-library atof() does. Ie only converting as much as it can and returning 0.0 for an error. """ match = _float_pattern.search(value) if match: return float(match.group(1)) return 0.0 >>> atof("15.5 Sausages") 15.5 >>> atof(" 17.2") 17.199 >>> atof("0x12") 0.0 >>> atof("8.3.2") 8.3007 >>> atof("potato") 0.0 >>> -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Found a very nice, small, cross-platform GUI toolkit for Python.
laplacia...@gmail.com wrote: > I think I just found the GUI toolkit for Python I've been searching > for. It seems to meet all of the following requirements: > >* free software >* small (I don't need batteries -- Python already comes with those.) >* easy to use >* actively maintained >* cross-platform >* easy to install >* based on a stable and actively-maintained C library >* does not depend on an external scripting language (for example, > Tcl) >* well-documented >* not too many dependencies >* can easily integrate with PyOpenGL >* support for accessibility > > and it's also written in Python. > > I have no idea how it's stayed under the radar in the Python community > for this long, yet here it is: [OcempGUI](http://ocemp.sourceforge.net/ > gui.html). The C library it depends upon? [SDL](http:// > www.libsdl.org/) (via [PyGame](http://www.pygame.org/news.html)). Interesting! One of the commercial apps I'm involved (C++ not python) in uses SDL as its GUI with windows etc built on top of it. It means that it looks exactly the same on all supported platforms and since it usually runs full screen that is fine. I imagine this GUI toolkit fits the same niche. Presumably since it uses SDL then all the GUI will appear in one window? So windows within windows in the MDI style? -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: python in emacs
I would go to ubuntu linux if you can. --- On Sun, 2/15/09, Diez B. Roggisch wrote: From: Diez B. Roggisch Subject: Re: python in emacs To: python-list@python.org Date: Sunday, February 15, 2009, 9:23 AM kentand...@sbcglobal.net schrieb: > When I visit a file with extension .py, emacs says "loading > python...done", and gives me a "python" menu with options like "start > interpreter" and "eval buffer". When I try to use one of these options > emacs says "loading compile...done", then hangs and has to be shut down > from the task manager. The Python folder is in my PATH variable and > works fine from the command line (or with IDLE). I'm using emacs-22.3 on > windows xp, and have the same problem with both python-2.2 and > python-3.0. Anybody got any tips, or should I just give up and use some > other editor? Did you try setting the python interpreter with the full path? Diez -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Easier to wrap C or C++ libraries?
Christian Heimes wrote: > Hrvoje Niksic schrieb: > > "Diez B. Roggisch" writes: > > > >> The answer is easy: if you use C, you can use ctypes to create a > >> wrapper - with pure python, no compilation, no platform issues. > > > > The last part is not true. ctypes doesn't work on 64-bit > > architectures, nor does it work when Python is built with non-gcc Unix > > compilers > > ctypes works on 64bit systems, too. However it's a pain in the ... back > to write code with ctypes that works across all platforms. I used to use > ctypes for wrapper but eventually I switched to Cython. What sort of problems have you had? I find as long as I use the same types as the C code actually uses it all works fine. If on a 64 bit platform long is 64 bits then it will be under ctypes too. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: encrypting lines from file with md5 module doesn't work?
Canned wrote: > I write a small script that read lines from plain text file and encrypt > each lines using md5 module. I have a small word list that contain 2000+ > words, 1 word/line. Using the code below, I can save the output to > another file to use it with john the ripper (http://www.openwall.com). > > Here is the part that annoys me, john recognize the output as des and > not as md5. Using the original wordlist, I tried to crack the list but > nothing has come up after waiting for almost 9 hours. Can someone please > tell me what did I wrong? Why john don't recognize the output as md5? > I'm using python 2.5.1 and I'm python noob and also don't have any > knowledge about encryption. > > import sys, md5 > > f = open(sys.argv[1]) > obj = md5.new() > > for line in f: > if line[-1:] == '\n': > text = line[:-1] > obj.update(text), > print text + ':' + obj.hexdigest() > > f.close() > > > 00:670b14728ad9902aecba32e22fa4f6bd > :c47532bbb1e2883c902071591ae1ec9b > 11:bf874003f752e86e6d6ba6d6df1f24a2 > :65a89de3000110bf37bcafdbd33df55a > 121212:38a8eeb4dfb0f86aefea908365817c15 > 123123:f226a65a908909b83aed92661897d0c9 john cracks password files if I remember rightly. md5 encoded password files don't look like that, they look like this guest:$1$3nvOlOaw$vRWaitT8Ne4sMjf9NOrVZ.:13071:0:9:7::: (not a real password line!) You need to work out how to write that format. >From memory: the "$1" bit means it is an md5 hash, the next "$3nvOlOaw$" is the salt and the final "$vRWaitT8Ne4sMjf9NOrVZ." is the md5 hash in some encoded format or other! Some googling should reveal the correct algorithm! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: String concatenation performance with +=
Sammo wrote: > String concatenation has been optimized since 2.3, so using += should > be fairly fast. > > In my first test, I tried concatentating a 4096 byte string 1000 times > in the following code, and the result was indeed very fast (12.352 ms > on my machine). > > import time > t = time.time() > mydata = "" > moredata = "A"*4096 > for i in range(1000): > mydata += moredata # 12.352 ms > print "%0.3f ms"%(1000*(time.time() - t)) > > However, I got a different result in my second test, which is > implemented in a class with a feed() method. This test took 4653.522 > ms on my machine, which is 350x slower than the previous test! > > class StringConcatTest: > def __init__(self): > self.mydata = "" > > def feed(self, moredata): > self.mydata += moredata # 4653.522 ms > > test = StringConcatTest() > t = time.time() > for i in range(1000): > test.feed(moredata) > print "%0.3f ms"%(1000*(time.time() - t)) > > Note that I need to do something to mydata INSIDE the loop, so please > don't tell me to append moredata to a list and then use "".join after > the loop. > > Why is the second test so much slower? The optimized += depends on their being no other references to the string. Strings are immutable in python. So append must return a new string. However the += operation was optimised to do an in-place append if and only if there are no other references to the string. You can see this demonstrated here $ python -m timeit -s 'a="x"' 'a+="x"' 100 loops, best of 3: 0.231 usec per loop $ python -m timeit -s 'a="x"; b=a' 's = a; a+="x"' 10 loops, best of 3: 30.1 usec per loop You are keeping the extra reference in a class instance like this $ python -m timeit -s 'class A(object): pass' -s 'a=A(); a.a="x"' 'a.a+="x"' 10 loops, best of 3: 30.7 usec per loop Knowing that, this optimization suggests itself $ python -m timeit -s 'class A(object): pass' -s 'a=A(); a.a="x"' 's = a.a; a.a = None; s += "x"; a.a = s' 100 loops, best of 3: 0.954 usec per loop Or in your example class StringConcatTest: def __init__(self): self.mydata = "" def feed(self, moredata): #self.mydata += moredata s = self.mydata del self.mydata s += moredata self.mydata = s moredata = "A"*4096 test = StringConcatTest() t = time.time() for i in range(1000): test.feed(moredata) print "%0.3f ms"%(1000*(time.time() - t)) Before it was 3748.012 ms on my PC, afterwards it was 52.737 ms However that isn't a perfect solution - what if something had another reference on self.mydata? You really want a non-immutable string for this use. array.array is a possibility $ python -m timeit -s 'import array' -s 'a = array.array("c")' 'a.extend("x")' 10 loops, best of 3: 2.01 usec per loop There are many other possibilities though like the mmap module. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Escaping my own chroot...
Jean-Paul Calderone wrote: > On Wed, 11 Feb 2009 09:31:56 -0600, Nick Craig-Wood > wrote: > >r0g wrote: > >> I'm writing a linux remastering script in python where I need to chroot > >> into a folder, run some system commands and then come out and do some > >> tidying up, un-mounting proc & sys etc. > >> > >> I got in there with os.chroot() and I tried using that to get back out > >> but that didn't work so... is my script trapped in there forever now or > >> is there an un-hacky way to escape? > > > >No! > > If you still have root in the chroot (and you need root to get in there, so > it's not implausible that you will), then you can get out. Googling for > "escape chroot" turns up lots of hits. This page contains a fairly simple, > explicit description of how to get out of a chroot: > > http://www.bpfh.net/simes/computing/chroot-break.html > > See the bulleted list in the "Breaking chroot()" section. Since you also > control the process before the chroot happens, breaking out is even simpler > in your case (just open / before you chroot in the first place). forking > before doing the chroot may still be a good idea, but it's not the only > solution. I admit it can be done, but I'm not sure it isn't hacky! #!/usr/bin/python """ Enter a chroot and escape again Run as root """ import os import sys def ls(path): """List the path""" print "Directory listing of %r" % path for f in os.listdir(path): print ">>", f def main(): if len(sys.argv) < 2: print >>sys.stderr, "Need directory to chroot to as an argument" raise SystemExit(1) chroot_dir = sys.argv[1] print "Opening root" root = os.open("/", os.O_RDONLY) print "Before chroot" ls("/") print "Chrooting to %r" % chroot_dir os.chroot(chroot_dir) ls("/") print "Breaking the chroot" os.fchdir(root) for i in range(100): os.chdir("..") os.chroot(".") ls("/") os.close(root) if __name__ == "__main__": main() I ran this $ mkdir chroot_test $ touch chroot_test/in_the_chroot $ sudo ./chroot_test.py chroot_test And it produced this Opening root Before chroot Directory listing of '/' >> lost+found >> home >> bin >> boot >> proc >> dev >> etc [snip] Chrooting to 'chroot_test' Directory listing of '/' >> in_the_chroot Breaking the chroot Directory listing of '/' >> lost+found >> home >> bin >> boot >> proc >> dev >> etc [snip] -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Escaping my own chroot...
r0g wrote: > I'm writing a linux remastering script in python where I need to chroot > into a folder, run some system commands and then come out and do some > tidying up, un-mounting proc & sys etc. > > I got in there with os.chroot() and I tried using that to get back out > but that didn't work so... is my script trapped in there forever now or > is there an un-hacky way to escape? No! > If it is an OS restriction rather than my ignorance of python is there a > canonical way of dealing with it? Should I maybe fork a process, have it > do the chroot and wait for it to terminate before continuing? That is the only way... However this is a solved problem if you use schroot (which works very well in my experience) http://packages.debian.org/sid/schroot http://www.debian-administration.org/articles/566 There are definitely debian, ubuntu and centos packages for it if you look! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: MySQLdb and MySQL stored functions
kurt.forrester@googlemail.com wrote: > Any ideas on how to suppress the warning output: > __main__:1: Warning: No data - zero rows fetched, selected, or > processed You can use the warnings module to suppress these I would have thought. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Cross platform compilation?
Christian Heimes wrote: > John Harper schrieb: > > I am trying to build Python to use in an embedded system which uses a > > ppc_440 CPU. The only information I've found is something written by > > Klaus Reimer a few years ago, which was based on Python 2.2. So far I > > seem to have successfully built Python itself, but building the > > extensions fails miserably with complaints about library format. It > > insists on referring to "i686" which is indeed the platform it is > > actually running on. > > > > Before I try to reverse engineer completely setup.py, is there > > something obvious that needs to be done to get it to use the right tool > > chain? I spent some time mucking about trying to do this for ARM. I almost got it to work using qemu but not quite! What I did in the end was to use a pre-compiled python (from debian) then copy the headers and library files into the cross compiling environment so we could embed python into our code and build our extensions. > > More generally, it seems like this would be something that lots of > > people would want to do. I'm surprised there isn't support for it built > > into the distributed version of Python, without having to hack all the > > setup files...? > > I'm sorry to inform you that Python doesn't support cross platform > compilation. Python eats its own dog food and uses Python to compile > optional extension modules. It isn't an impossible problem... If you've ever cross compiled gcc you'll find that it first uses the host compiler (possibly not gcc) to compile itself to make a gcc which runs on the host but makes code for the target. It then uses that new compiler to compile a gcc for the target architecture. I could imagine a similar scheme for python, but it would involve lots of fiddling about and some commitment from the python maintainers. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Code critique xmlrpclib
flagg wrote: > On Feb 3, 7:32?am, Nick Craig-Wood wrote: > > flagg wrote: > > > ?This xmlrpc server is designed to parse dns zone files and then > > > ?perform various actions on said files. \ > > > ?It uses dnspython, and xmlrpclib > > > ? I'd like to know what some of the more experienced python users > > > ?think. Where I could improve code, make it more efficient, whatever. > > > ?All suggestions are welcome. ?This is my first functional python > > > ?program, ?and I have tons of questions to go along with whatever > > > ?suggestions. ? How could I do sanity checking; right now nothing > > > ?checks the input to these functions, error-handling in this program is > > > ?almost non-existant how do you integrate decent error handling into a > > > ?piece of software..ill shut up now. > > > > I made a few comments, some about error handling! ?See below > > > > > > > > > ?import dns.zone > > > ?from time import localtime, strftime, time > > > ?import os, sys > > > ?from dns.rdataclass import * > > > ?from dns.rdatatype import * > > > ?from string import Template > > > ?import xmlrpclib > > > ?from SimpleXMLRPCServer import SimpleXMLRPCServer > > > ?from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler > > > > > ?zoneDir = "/dns" > > > > > ?def openZoneFile(zoneFile): > > > ? ? ?""" > > > ? ? ?Opens a zone file. ?Then reads in zone file to dnspython zone > > > ?object. > > > ? ? ?""" > > > ? ? ?global zone > > > ? ? ?global host > > > ? ? ?global domain > > > ? ? ?domain = ".".join(zoneFile.split('.')[1:]) > > > ? ? ?host = ".".join(zoneFile.split('.')[:1]) > > > > > ? ? ?try: > > > ? ? ? ? ? ? ?zone = dns.zone.from_file(zoneDir + domain + '.zone', > > > ?domain) > > > ? ? ?except dns.exception, e: > > > ? ? ? ? ? ? ?print e.__class__, e > > > > Don't use global variables! > > > > This function should probably return 3 things > > > > ? ? return zone, host, domain > > > > And whenever you use it, say > > > > ? ? zone, host, domain = openZoneFile(xxx) > > > > It should probably be a method of DNSFunctions. ?I'd avoid setting > > self.zone etc as that is making a different sort of global data which > > I'd avoid too. > > > > Also if the dns.zone.from_file didn't work (raises dns.exception) you > > set the host and domain but the zone is left at its previous value > > which probably isn't what you wanted. ?I'd let the error propagate > > which I think will do something sensible for the xmlrpc. > > > > > > > > > ?class DNSFunctions: > > > > > ? ? ? # Not exposed to xml-rpc calls, used internally only. > > > ? ? ?def _writeToZoneFile(self, record): > > > ? ? ? ? ?""" > > > ? ? ? ? ?Checks if zone file exists, if it does it write values to zone > > > ?file > > > ? ? ? ? ?""" > > > > > ? ? ? ? ?for (name, ttl, rdata) in zone.iterate_rdatas(SOA): > > > ? ? ? ? ? ? ?new_serial = int(strftime('%Y%m%d00', localtime(time( > > > ? ? ? ? ? ? ?if new_serial <= rdata.serial: > > > ? ? ? ? ? ? ? ? ?new_serial = rdata.serial + 1 > > > ? ? ? ? ? ? ?rdata.serial = new_serial > > > > > ? ? ? ? ?if os.path.exists(zoneDir + str(zone.origin) + "zone"): > > > ? ? ? ? ? ? ?f = open(zoneDir + str(zone.origin) + "zone", "w") > > > ? ? ? ? ? ? ?zone.to_file(f) > > > ? ? ? ? ? ? ?f.close() > > > ? ? ? ? ?else: > > > ? ? ? ? ? ? ?print "Zone: " + zone.origin + " does not exist, please > > > ?add first" > > > > This should probably be raising an exception to be caught or > > propagated back to the client via xmlrpc. > > > > > > > > > > > > > ? ? ?def showRecord(self, record): > > > ? ? ? ? ?""" > > > ? ? ? ? ?Shows a record for a given zone. Prints out > > > ? ? ? ? ?TTL, IN, Record Type, IP > > > ? ? ? ? ?""" > > > > > ? ? ? ? ?openZoneFile(record) > > > > > ? ? ? ? ?try: > > > ? ? ? ? ? ? ?rdataset = zone.get_node(host) > > > ? ? ? ? ? ? ?for rdata in rdataset: > > > ? ? ? ? ? ? ? ? ?return str(rdata) > > > ? ? ? ? ?except: > >
Re: Code critique xmlrpclib
flagg wrote: > This xmlrpc server is designed to parse dns zone files and then > perform various actions on said files. \ > It uses dnspython, and xmlrpclib > I'd like to know what some of the more experienced python users > think. Where I could improve code, make it more efficient, whatever. > All suggestions are welcome. This is my first functional python > program, and I have tons of questions to go along with whatever > suggestions. How could I do sanity checking; right now nothing > checks the input to these functions, error-handling in this program is > almost non-existant how do you integrate decent error handling into a > piece of software..ill shut up now. I made a few comments, some about error handling! See below > import dns.zone > from time import localtime, strftime, time > import os, sys > from dns.rdataclass import * > from dns.rdatatype import * > from string import Template > import xmlrpclib > from SimpleXMLRPCServer import SimpleXMLRPCServer > from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler > > > zoneDir = "/dns" > > def openZoneFile(zoneFile): > """ > Opens a zone file. Then reads in zone file to dnspython zone > object. > """ > global zone > global host > global domain > domain = ".".join(zoneFile.split('.')[1:]) > host = ".".join(zoneFile.split('.')[:1]) > > try: > zone = dns.zone.from_file(zoneDir + domain + '.zone', > domain) > except dns.exception, e: > print e.__class__, e Don't use global variables! This function should probably return 3 things return zone, host, domain And whenever you use it, say zone, host, domain = openZoneFile(xxx) It should probably be a method of DNSFunctions. I'd avoid setting self.zone etc as that is making a different sort of global data which I'd avoid too. Also if the dns.zone.from_file didn't work (raises dns.exception) you set the host and domain but the zone is left at its previous value which probably isn't what you wanted. I'd let the error propagate which I think will do something sensible for the xmlrpc. > class DNSFunctions: > > # Not exposed to xml-rpc calls, used internally only. > def _writeToZoneFile(self, record): > """ > Checks if zone file exists, if it does it write values to zone > file > """ > > for (name, ttl, rdata) in zone.iterate_rdatas(SOA): > new_serial = int(strftime('%Y%m%d00', localtime(time( > if new_serial <= rdata.serial: > new_serial = rdata.serial + 1 > rdata.serial = new_serial > > if os.path.exists(zoneDir + str(zone.origin) + "zone"): > f = open(zoneDir + str(zone.origin) + "zone", "w") > zone.to_file(f) > f.close() > else: > print "Zone: " + zone.origin + " does not exist, please > add first" This should probably be raising an exception to be caught or propagated back to the client via xmlrpc. > > def showRecord(self, record): > """ > Shows a record for a given zone. Prints out > TTL, IN, Record Type, IP > """ > > openZoneFile(record) > > try: > rdataset = zone.get_node(host) > for rdata in rdataset: > return str(rdata) > except: > raise Exception("Record not found") > > > def changeRecord(self, record, type, target): > """ > Changes a dns entry. > @param record: which record to chance > @param type: what type of record, A, MX, NS, CNAME > @target: if CNAME target points to HOSTNAME, if A record > points to IP > """ > openZoneFile(record) > try: > rdataset = zone.find_rdataset(host, rdtype=type) > except: > raise Exception("You must enter a valid record and type. > See --usage for examples") Don't catch all exceptions - find out which exceptions are thrown. Consider just letting it propagate - hopefully the find_rdataset error is descriptive enough. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Does the Python community really follow the philospy of "Community Matters?"
I think what you have found is a remarkable characteristic of this language. Somehow, perhaps something to do with guido or python itself, python has a very strong non-dogmatic streak. It's a relief really. If I were to pose a "is python or its community really xyz?" I would wonder about the "one best way to do something". One of my first posts here was about three different ways I had built singletons. As a result of that I was given a better way, but no one said any of the other ways were "non-pythonic", it was more, "if you use option one, make sure of this, if you use option two, don't forget that"... As such, python programmers are not there for you to feel superior. If you want a language for a feeling of community, then it hardly matters what language you choose, and if you want a good language, it's secondary if the community gives you that feeling (you should get it from the language itself). And since there IS a python community to go to for help, python has that covered without worrying about it or making it a language feature. It may seems cold, instead of giving warm fuzzies, people will help you with tangible problems, but we are not starting a commune. "Uses a langague with sense of community that advocates for their language over others" is never in a spec. -craig -- http://mail.python.org/mailman/listinfo/python-list
Re: what IDE is the best to write python?
eclipse --- On Sun, 2/1/09, Dennis Lee Bieber wrote: From: Dennis Lee Bieber Subject: Re: what IDE is the best to write python? To: python-list@python.org Date: Sunday, February 1, 2009, 3:31 AM On Sat, 31 Jan 2009 23:42:42 -0800 (PST), "mcheun...@hotmail.com" declaimed the following in comp.lang.python: > Hi all > what IDE is the best to write python? The one in which YOU are most comfortable. -- Wulfraed Dennis Lee Bieber KD6MOG wlfr...@ix.netcom.com wulfr...@bestiaria.com HTTP://wlfraed.home.netcom.com/ (Bestiaria Support Staff: web-a...@bestiaria.com) HTTP://www.bestiaria.com/ -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.6's multiprocessing lock not working on second use?
Jesse Noller wrote: > > Opened issue #4999 [http://bugs.python.org/issue4999] on the matter, > > referencing this thread. > > Thanks, I've assigned it to myself. Hopefully I can get a fix put > together soonish, time permitting. Sounds like it might be hard or impossible to fix to me. I'd love to be proved wrong though! If you were thinking of passing time.time() / clock_gettime(CLOCK_MONOTONIC) along in the Queue too, then you'll want to know that it can differ by significant amounts on different processors :-( Good luck! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.6's multiprocessing lock not working on second use?
Gabriel Genellina wrote: > En Fri, 16 Jan 2009 14:41:21 -0200, escribiste en el grupo > gmane.comp.python.general > > > I ran a few tests on the new Python 2.6 multiprocessing module before > > migrating a threading code, and found out the locking code is not > > working well. In this case, a pool of 5 processes is running, each > > trying to get the lock and releasing it after waiting 0.2 seconds > > (action is repeated twice). It looks like the multiprocessing lock > > allows multiple locking after the second pass. Running the exact same > > code with threads works correctly. > > I've tested your code on Windows and I think the problem is on the Queue > class. If you replace the Queue with some print statements or write to a > log file, the sequence lock/release is OK. > You should file a bug report on http://bugs.python.org/ A little bit of instrumentation in the code reveals the problem. The Queue class doesn't always return the items in the order that they were put in. This should probably be either documented or fixed! I suspect it is impossible to fix for a multi-producer Queue though. The first number is time since the program started. Tested under linux. 0.048810 [proc0] Got lock 0.248679 [proc0] Released lock 0.248858 [proc0] Got lock 0.448666 [proc0] Released lock 0.448859 [proc2] Got lock 0.648639 [proc2] Released lock 0.648893 [proc3] Got lock 0.848633 [proc3] Released lock 0.848767 [proc3] Got lock 1.048635 [proc3] Released lock 1.049090 [proc1] Got lock 1.248617 [proc1] Released lock 1.248743 [proc1] Got lock 1.448634 [proc1] Released lock 1.448810 [proc4] Got lock 1.648674 [proc4] Released lock 1.648831 [proc4] Got lock 1.849867 [proc2] Got lock<--- out of order 1.849679 [proc4] Released lock <--- out of order 2.048683 [proc2] Released lock #!/usr/bin/python -*- coding: utf-8 -*- from multiprocessing import Process, Queue, Lock from Queue import Empty from threading import Thread import time start = time.time() def now(): return time.time() - start class test_lock_process(object): def __init__(self, lock, id, queue): self.lock = lock self.id = id self.queue = queue self.read_lock() def read_lock(self): for i in xrange(2): self.lock.acquire() self.queue.put('%9.6f [proc%d] Got lock' % (now(), self.id)) time.sleep(.2) self.queue.put('%9.6f [proc%d] Released lock' % (now(), self.id)) self.lock.release() def test_lock(processes=10, lock=Lock(), process=True, queue=None): print_result = False if queue == None: print_result = True queue = Queue() threads = [] for i in xrange(processes): if process: threads.append(Process(target=test_lock_process, args=(lock,i,queue,))) else: threads.append(Thread(target=test_lock_process, args=(lock,i,queue,))) for t in threads: t.start() for t in threads: t.join() if print_result: try: while True: print queue.get(block=False) except Empty: pass if __name__ == "__main__": #test_lock(processes=5, process=True) test_lock(processes=5) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: initialising a class by name
Krishnakant wrote: > I liked this idea of dispatchTable. > is it possible to say some thing like > inst = dispatchTable{"ham"} > according to me, inst will become the instance of class ham. > Another thing to note is that all the classes are in different modules. > So where do I create the dict of classes mapped with the name? You could use a metaclass which will get round the problem, provided all the classes have a common baseclass, eg class BaseThing(object): registry = {} class __metaclass__(type): def __init__(cls, name, bases, dict): cls.registry[cls.__name__] = cls class ThingOne(BaseThing): pass class ThingTwo(BaseThing): pass class ThingThree(BaseThing): pass print BaseThing.registry["ThingOne"] print BaseThing.registry["ThingTwo"] print BaseThing.registry["ThingThree"] Which prints -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: ctype problem
Grimson wrote: > hello out there, > I have a problem with c-types. > I made a c-library, which expects a pointer to a self defined structure. > > let the funtion call myfunction(struct interface* iface) > > and the struct: > struct interface > { > int a; > int b; > char *c; > } > > the Python ctype port of this structur would be: > > class INTERFACE(Structure): > _fields_ = [("a" c_int), >("b",c_int), >("c", c_char)] > > in my python-struct a create a instance of INTERFACE > > myiface = INTERFACE() > myiface.a = ctypes.c_int(80) > myiface.b = ctypes.c_int(22) > ... > than I make a pointer onto it. > p_iface = ctypes.pointer(myiface) > and I tried it also with a reference > r_iface = ctypes.byref(myiface) > > but neither myclib.myfunction(p_iface) nor myclib.myfunction(r_iface) > works properly. The function is been called but it reads only zeros (0) > for each parameter (member in the struct). > > Where is my fault? You didn't (or you didn't show) defining the argument types of the function. myclib = CDLL("myclib.so") # or whatever myclib.myfunction.argtypes = [ POINTER(INTERFACE) ] myclib.myfunction.restype = c_int # or whatever If you do that then you should be able to pass in myiface directly or byref(myiface). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Python tricks
Scott David Daniels wrote: > RajNewbie wrote: > > On Jan 12, 6:51 pm, Tim Chase wrote: > [a perfectly fine reply which is how I'd solve it] > >> RajNewbie wrote: > >>> ... The solution that I had in mind is: > >>>while True: > >>> ... > >>> if : break > >>> if inifinte_loop(): raise infiinte_loop_exception > >>> Wherein infinite_loop is a generator, which returns true if i > 200 > >>> def infinite_loop(): > >>> i = 0 > >>> while i < 200: > >>> i++ > >>> yield False > >>> yield True > >>> Could somebody let me know whether this is a good option? > > ... > > But, I still feel it would be much more aesthetically pleasing if I > > can call a single procedure like > > if infinite_loop() -> to do the same. > > Is it somehow possible? - say by using static variables, iterators -- > > anything? > > Yes, it is possible. After: > > def Fuse(count, exception): > for i in range(count): > yield None > raise exception > > You can do your loop as: > check_infinite = Fuse(200, ValueError('Infinite Loop')).next > while True: > ... > check_infinite() Or related to the above and the original proposal class InfiniteLoopError(Exception): """An 'infinite' loop has been detected""" def infinite_loop(max=200): for i in xrange(max): yield i raise InfiniteLoopError() Use it like this for i in infinite_loop(): if i > 10: break print "iteration", i or for i in infinite_loop(10): print "iteration", i > but I agree with Tim that a for ... else loop for the limit is > clearer. Probably yes -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Extending Python with C or C++
Thomas Heller wrote: > Nick Craig-Wood schrieb: > > Thomas Heller wrote: > >> Nick Craig-Wood schrieb: > >> > Interesting - I didn't know about h2xml and xml2py before and I've > >> > done lots of ctypes wrapping! Something to help with the initial > >> > drudge work of converting the structures would be very helpful. > >> > > >> > ( http://pypi.python.org/pypi/ctypeslib/ ) > >> > > >> > I gave it a quick go and it worked fine. I had to edit the XML in one > >> > place to make it acceptable (removing a u from a hex number). > >> > >> If you are using a recent version of gccxml, then you should use the > >> ctypeslib-gccxml-0.9 branch of ctypeslib instead of the trunk. (I should > >> really merge the gccxml-0.9 branch into the trunk;-) > >> > >> If you are already using the branch and the XML file is not accepted, > >> then could you please provide a short C code snippet that reproduces > >> the problem so that I can fix it? > > > > I'm using gccxml from debian testing 0.9.0+cvs20080525-1 which I guess > > doesn't have the code from the branch in. > > I meant the branch in the repository where ctypeslib lives in. > Anyway, it doesn't matter anymore since I merged that branch > into the ctypeslib trunk. > > Now the problem with the 'u' suffix that you mentioned should be fixed, and > I also made a quick change so that the sized integer types from stdint.h are > generated > correctly. So, please update your ctypeslib installation and trey again;-) I gave it a go and I can report success on both counts! Well done and thank you. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Creating new instances of subclasses.
J. Cliff Dyer wrote: > I want to be able to create an object of a certain subclass, depending > on the argument given to the class constructor. > > I have three fields, and one might need to be a StringField, one an > IntegerField, and the last a ListField. But I'd like my class to > delegate to the proper subclass automatically, so I can just do: > > >>> f1 = Field('abc') > >>> f2 = Field('123') > >>> f3 = Field('D,E,F') > >>> f1.data > 'abc' > >>> f2.data > 123 > >>> f3.data > ['D','E','F'] > >>> type(f1) > > >>> type(f2) > > >>> type(f3) > > > I've come up with a solution, but I suspect there's something cleaner I > can do with the inheritance structure of __new__. I don't like > explicitly leapfrogging over Field.__new__ to object.__new__. > > My attempt is below: > > def is_list(arg): > if ',' in arg: return True > else: return False > > def is_integer(arg): > try: int(arg) > except ValueError: return False > else: return True > > class Field(object): > def __new__(cls, a): > if is_list(a): > return ListField(a) > elif is_integer(a): > return IntegerField(a) > else: > return StringField(a) > > def __init__(self, input): > super(Field, self).__init__(input) > self.data = input > > class IntegerField(Field): > def __new__(cls, a): > return object.__new__(cls, a) > def __init__(self, s): > super(IntegerField, self).__init__(s) > self.s = int(self.s) > > class ListField(Field): > def __new__(cls, a): > return object.__new__(cls, a) > def __init__(self, s): > super(ListField, self).__init__(s) > self.s = s.split(',') > > class StringField(Field): > def __new__(cls, a): > return object.__new__(cls, a) > > Is there a cleaner way to do this? The main problem is that > Field.__new__ gets in the way of properly constructing the subclasses > once I've used it to select the proper subclass in the first place. How about this minor modification? # rest as above class Field(object): def __new__(cls, a): if cls != Field: return object.__new__(cls, a) if is_list(a): return ListField(a) elif is_integer(a): return IntegerField(a) else: return StringField(a) def __init__(self, input): super(Field, self).__init__(input) self.data = input class IntegerField(Field): def __init__(self, s): super(IntegerField, self).__init__(s) self.s = int(s) class ListField(Field): def __init__(self, s): super(ListField, self).__init__(s) self.s = s.split(',') class StringField(Field): pass Or you could go for the full metaclass self registration scheme like this, which is actually less code since I delegated the "is this ok" test to the subclass - failing it returns a ValueError. class Field(object): registry = [] class __metaclass__(type): def __init__(cls, name, bases, dict): cls.registry.append(cls) def __new__(cls, a): if cls != Field: return object.__new__(cls, a) for subcls in cls.registry: if subcls == Field: continue try: return subcls(a) except ValueError: pass raise ValueError("Couldn't find subclass") def __init__(self, input): super(Field, self).__init__(input) self.data = input # Raise a ValueError in init if not suitable args for this subtype class IntegerField(Field): def __init__(self, s): s = int(s) super(IntegerField, self).__init__(s) self.s = s class ListField(Field): def __init__(self, s): if ',' not in s: raise ValueError("Not a list") super(ListField, self).__init__(s) self.s = s.split(',') class StringField(Field): pass -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing takes higher execution time
Sibtey Mehdi wrote: > I use multiprocessing to compare more then one set of files. > > For comparison each set of files (i.e. Old file1 Vs New file1) > I create a process, > > Process(target=compare, args=(oldFile, newFile)).start() > > It takes 61 seconds execution time. > > When I do the same comparison without implementing > multiprocessing, it takes 52 seconds execution time. > The oldProjects and newProjects will contains zip files > i.e(oldxyz1.zip,oldxyz2.zip, newxyz2.zip,newxyz2.zip) > it will unzip both the zip files and compare all the files between old > and new (mdb files or txt files) and gives the result. > I do this comparision for n number set of zip files and i assigne each > set of zip files comparision to a process. I had a brief look at the code and your use of multiprocessing looks fine. How many projects are you processing at once? And how many MB of zip files is it? As reading zip files does lots of disk IO I would guess it is disk limited rather than anything else, which explains why doing many at once is actually slower (the disk has to do more seeks). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing takes higher execution time
Grant Edwards wrote: > On 2009-01-07, Steve Holden wrote: > > >> I use multiprocessing to compare more then one set of files. > >> > >> For comparison each set of files (i.e. Old file1 Vs New file1) > >> I create a process, > >> > >> Process(target=compare, args=(oldFile, newFile)).start() > >> > >> It takes 61 seconds execution time. > >> > >> When I do the same comparison without implementing > >> multiprocessing, it takes 52 seconds execution time. > > > My first suggestion would be: show us some code. We aren't > > psychic, you know. > > I am! > > He's only got one processor, and he's just been bit by Amdahl's > law when P<1 and S<1. > > There you have a perfectly "psychic" answer: an educated guess > camoflaged in plausible-sounding but mostly-bullshit buzzwords. > A better psychic would have avoided making that one falsifiable > statement (he's only got one processor). ;-) My guess would be that the job is IO bound rather than CPU bound, but that is covered by Amdahl's Law too where P is approx 0, N irrelevant... Being IO bound explains why it takes longer with multiprocessing - it causes more disk seeks to run an IO bound algorithm in parallel than running it sequentially. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Extending Python with C or C++
Thomas Heller wrote: > Nick Craig-Wood schrieb: > > Interesting - I didn't know about h2xml and xml2py before and I've > > done lots of ctypes wrapping! Something to help with the initial > > drudge work of converting the structures would be very helpful. > > > > ( http://pypi.python.org/pypi/ctypeslib/ ) > > > > I gave it a quick go and it worked fine. I had to edit the XML in one > > place to make it acceptable (removing a u from a hex number). > > If you are using a recent version of gccxml, then you should use the > ctypeslib-gccxml-0.9 branch of ctypeslib instead of the trunk. (I should > really merge the gccxml-0.9 branch into the trunk;-) > > If you are already using the branch and the XML file is not accepted, > then could you please provide a short C code snippet that reproduces > the problem so that I can fix it? I'm using gccxml from debian testing 0.9.0+cvs20080525-1 which I guess doesn't have the code from the branch in. > > Here are my thoughts on the conversion :- > > > > It converted an interface which looked like this (the inotify interface) > > > > struct inotify_event { > > int wd; /* Watch descriptor */ > > uint32_t mask; /* Mask of events */ > > uint32_t cookie; /* Unique cookie associating related > > events (for rename(2)) */ > > uint32_t len; /* Size of name field */ > > char name[]; /* Optional null-terminated name */ > > }; > > > > Into this > > > > class inotify_event(Structure): > > pass > > inotify_event._fields_ = [ > > ('wd', c_int), > > ('mask', uint32_t), > > ('cookie', uint32_t), > > ('len', uint32_t), > > ('name', c_char * 0), > > ] > > > > Which is a very good start. However it defined these which clearly > > aren't portable > > > > int32_t = c_int > > uint32_t = c_uint > > > > Whereas it should have been using the ctypes inbuilt types > > > > c_int32 > > c_uint32 > > IMO this would be difficult to achive automatically. There are cases > wher c_int is the correct type, in other cases c_int32 is correct. Yes it is almost impossible difficult to achieve automatically - exactly how far do you want to unravel the twisty turny mess of typedefs that make up uint32_t? It is easy to change the generated output since it defines the type in one place. uint32_t and friends (stdint.h) are standardised in C99 so might it be reasonable to put some special cases in for them, expecially since the corresponding types already exist in ctypes? -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Extending Python with C or C++
Ralf Schoenian wrote: > Ryan wrote: > > I've been using Python for many years now. It's a wonderful language > > that I enjoy using everyday. I'm now interested in getting to know > > more about the guts (C/C++) and extending it. But, extending python > > still seems like a black art to me. Is there anymore docs or info on > > extending it besides the standard sparse ones (http://www.python.org/ > > doc/2.5.2/ext/intro.html) that may give me more insight? Is there a > > class available? How can I learn more about the guts of python? How > > would one go about following an interest in contributing to the > > development of python. > > It is not exactly what you are looking for but nevertheless I am > thinking the article "Automatic C Library Wrapping -- Ctypes from the > Trenches" may be interesting for you. You can find it in the latest > Python Papers issue or simply following the link: > http://ojs.pythonpapers.org/index.php/tpp/article/view/71 Interesting - I didn't know about h2xml and xml2py before and I've done lots of ctypes wrapping! Something to help with the initial drudge work of converting the structures would be very helpful. ( http://pypi.python.org/pypi/ctypeslib/ ) I gave it a quick go and it worked fine. I had to edit the XML in one place to make it acceptable (removing a u from a hex number). The output of xml2py was an excellent place to start for the conversion, though I don't think I'd want to use an automated process like in the paper above as its output needed tweaking. ... Here are my thoughts on the conversion :- It converted an interface which looked like this (the inotify interface) struct inotify_event { int wd; /* Watch descriptor */ uint32_t mask; /* Mask of events */ uint32_t cookie; /* Unique cookie associating related events (for rename(2)) */ uint32_t len; /* Size of name field */ char name[]; /* Optional null-terminated name */ }; Into this class inotify_event(Structure): pass inotify_event._fields_ = [ ('wd', c_int), ('mask', uint32_t), ('cookie', uint32_t), ('len', uint32_t), ('name', c_char * 0), ] Which is a very good start. However it defined these which clearly aren't portable int32_t = c_int uint32_t = c_uint Whereas it should have been using the ctypes inbuilt types c_int32 c_uint32 Also I don't think c_char * 0 does anything sensible in ctypes, c_byte * 0 is what is required plus a bit of casting. This is a non-standard GNU extension to C though. All that said though, it looks like a great time saver. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing vs thread performance
mk wrote: > After reading http://www.python.org/dev/peps/pep-0371/ I was under > impression that performance of multiprocessing package is similar to > that of thread / threading. However, to familiarize myself with both > packages I wrote my own test of spawning and returning 100,000 empty > threads or processes (while maintaining at most 100 processes / threads > active at any one time), respectively. > > The results I got are very different from the benchmark quoted in PEP > 371. On twin Xeon machine the threaded version executed in 5.54 secs, > while multiprocessing version took over 222 secs to complete! > > Am I doing smth wrong in code below? Yes! The problem with your code is that you never start more than one process at once in the multiprocessing example. Just check ps when it is running and you will see. My conjecture is that this is due to the way fork() works under unix. I think that when the parent forks it yields the CPU to the child. Because you are giving the child effectively no work to do it returns immediately, re-awakening the parent, thus serialising your jobs. If you give the children some work to do you'll see a quite different result. I gave each child time.sleep(1) to do and cut down the total number to 10,000. $ ./test_multiprocessing.py == Process 1000 working == == Process 2000 working == == Process 3000 working == == Process 4000 working == == Process 5000 working == == Process 6000 working == == Process 7000 working == == Process 8000 working == == Process 9000 working == == Process 1 working == === Main thread waiting for all processes to finish === Total time: 101.382129192 $ ./test_threading.py == Thread 1000 working == == Thread 2000 working == == Thread 3000 working == == Thread 4000 working == == Thread 5000 working == == Thread 6000 working == == Thread 7000 working == == Thread 8000 working == == Thread 9000 working == == Thread 1 working == Total time: 100.659118176 So almost identical results and as expected - we ran 10,000 sleep(1)s in 100 seconds so we must have been running 100 simultaneously. If you replace the "time.sleep(1)" with "for _ in xrange(100): pass" you get this much more interesting answer on my dual core linux laptop, showing nicely the effect of the contention on the python global interpreter lock and how multiprocessing avoids it. $ ./test_multiprocessing.py == Process 1000 working == == Process 2000 working == == Process 3000 working == == Process 4000 working == == Process 5000 working == == Process 6000 working == == Process 7000 working == == Process 8000 working == == Process 9000 working == == Process 1 working == === Main thread waiting for all processes to finish === Total time: 266.808327913 $ ./test_threading.py == Thread 1000 working == == Thread 2000 working == == Thread 3000 working == == Thread 4000 working == == Thread 5000 working == == Thread 6000 working == == Thread 7000 working == == Thread 8000 working == == Thread 9000 working == == Thread 1 working == Total time: 834.81882 -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: I always wonder ...
this is one of the most subtle trolls I've ever read. you sir, are a master! On Dec 22, 7:53 am, s...@pobox.com wrote: > ... shouldn't people who spend all their time trolling be doing something > else: studying, working, writing patches which solve the problems they > perceive to exist in the troll subject? Is there some online troll game > running where the players earn points for generating responses to their > posts? > > -- > Skip Montanaro - s...@pobox.com -http://smontanaro.dyndns.org/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Python is slow
On Dec 14, 6:38 pm, cm_gui wrote: > hahaha, do you know how much money they are spending on hardware to > make > youtube.com fast??? > > > By the way... I know of a very slow Python site called YouTube.com. In > > fact, it is so slow that nobody ever uses it. less than they'd spend to implement it in C -- http://mail.python.org/mailman/listinfo/python-list
Re: How to modify a program while it's running?
On Dec 16, 10:25 am, Joe Strout wrote: > Here's my situation: I'm making an AIM bot, but the AIM server will > get annoyed if you log in too frequently (and then lock you out for a > while). So my usual build-a-little, test-a-little methodology doesn't > work too well. > > So I'd like to restructure my app so that it can stay running and stay > logged in, yet I can still update and reload at least most of the > code. But I'm not sure what's the best way to do this. Should I move > the reloadable code into its own module, and then when I give my bot a > "reload" command, have it call reload on that module? Will that work, > and is there a better way? > > Thanks, > - Joe yes you want reload. Design the high level part that knows how to log in to be able to reload the stuff that changes. My guess is that is the best way, though I wouldn't be surprised if there are other solutions. -- http://mail.python.org/mailman/listinfo/python-list
Re: How do I manually uninstall setuptools (installed by egg)?
David Cournapeau wrote: > On Thu, Dec 11, 2008 at 10:30 PM, Nick Craig-Wood > wrote: > > David Cournapeau wrote: > >> On Wed, Dec 10, 2008 at 12:04 PM, Chris Rebert wrote: > >> > On Tue, Dec 9, 2008 at 6:49 PM, wrote: > >> >> On Ubuntu, I accidentally manually installed setuptools > >> >> http://pypi.python.org/pypi/setuptools/0.6c9 (by running the .egg file > >> >> as a shell script via sudo), and now realize I should just be using > >> >> apt to take care of my system Python packages. > >> > > >> > Really, why? setuptools has more Python packages/programs available > >> > and updates faster than Debian. > >> > It's also likely that some of the Debian Python packages are installed > >> > using setuptools anyway. > >> > So, why do you think apt and not setuptools is The Right Way(tm)? > >> > >> Setuptools is certainly not the right way to install packages > >> system-wide on debian, it is very likely to break the whole thing. > > > > It wouldn't be too difficult to make a .deb target which would collect > > all the files that did get installed into a package. It would be a > > rather rough and ready package but would do the job. > > Depends what you mean by would do the job: rather rough certainly does > not mean "would do the job" for something as essential as a package > IMO. Essentially a package has files in it in a fixed possition in the filesystem. The package manager's (dpkg at this level) job is to keep track of those file and tell you about conflicts. You can use alien to turn a tar.gz into a perfectly usable debian package. It won't have dependencies, or help or any of the other things a package needs to be a proper package, but it satisfies my basic needs that all software is installed as packages, so it can be uninstalled cleanly. > > The .deb would then be uninstallable in the usual (dpkg --purge) way. > > > > Did anyone think about that? > > Yes, there is stddeb which does that (make a .deb package from a > setuptools package). Cool, I've not seen that before. Probably because it isn't in debian! > > easy_install can do that I think... > > Not without a lot of hoola, unfortunately; for example, it breaks > stow, so I have to use specialy scripts to circumvent the damn thing > and make it what I tell him to do. I never understood what's easy > about easy install: it is broken in so many ways, and has caused me so > many pains - even when I was not using - that I do not trust it at > all. I only use it to download packages (and even then it manage to > fail more than work), and always install them from setup.py afterwards > (at which step I of course also have to circumvent setuptools if the > package uses setuptools). ;-) As a mostly linux developer I always install stuff in packages if possible. Failing that I'll use bdist_rpm then alien to convert to a deb which works well enough I find. However when I have to make my stuff work on Windows, I find easy_install to be a fantastic timesaver as compared to looking for the package on a web site, downloading it, unpacking it installing it and then repeating for all the dependencies. easy_install is a long way from being a proper package manager though :-( I guess it is aiming at the same territory as the cpan installer with perl. My experiences with that is that it works very well, but again splatters untracked files all over your filesystem. Debian comes with dh-make-perl which can use CPAN directly to make .debs which works quite well. Something similar based on easy_install for python would be ideal. It looks like stdeb could become that one day. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python.NET] Where does the clr in IronPython look the dll
The dll needs to be on the Python path (sys.path). You can either add to the path with sys.path.append("c:\") or put your dll in a folder in the Python site-packages directory and add a .pth file (for Python.NET, but not IronPython -- it doesn't recognise the .pth files). Regards, Craig. 3/12/2008 9:20 p.m. dï, navneet khanna wrote: Hello Everybody I am trying to import dll with clr.AddReference("TCdll") I am getting the following error. Traceback (most recent call last): File "", line 1, in clr.AddReference("TCdll") FileNotFoundException: Unable to find assembly 'TCdll'. at Python.Runtime.CLRModule.AddReference(String name) I have loaded the dll in system32 folder. Have I loaded the dll at the right place or do I need to place it anywhere else? Waiting for your replies. Regards Navneet _ Python.NET mailing list - pythondot...@python.org http://mail.python.org/mailman/listinfo/pythondotnet -- http://mail.python.org/mailman/listinfo/python-list
Re: How do I manually uninstall setuptools (installed by egg)?
David Cournapeau <[EMAIL PROTECTED]> wrote: > On Wed, Dec 10, 2008 at 12:04 PM, Chris Rebert <[EMAIL PROTECTED]> wrote: > > On Tue, Dec 9, 2008 at 6:49 PM, <[EMAIL PROTECTED]> wrote: > >> On Ubuntu, I accidentally manually installed setuptools > >> http://pypi.python.org/pypi/setuptools/0.6c9 (by running the .egg file > >> as a shell script via sudo), and now realize I should just be using > >> apt to take care of my system Python packages. > > > > Really, why? setuptools has more Python packages/programs available > > and updates faster than Debian. > > It's also likely that some of the Debian Python packages are installed > > using setuptools anyway. > > So, why do you think apt and not setuptools is The Right Way(tm)? > > Setuptools is certainly not the right way to install packages > system-wide on debian, it is very likely to break the whole thing. It wouldn't be too difficult to make a .deb target which would collect all the files that did get installed into a package. It would be a rather rough and ready package but would do the job. The .deb would then be uninstallable in the usual (dpkg --purge) way. Did anyone think about that? There is a bdist_rpm target for distutils which I often use then use alien to turn into a .deb. easy_install is a lot easier though! > dpkg is a real package installer, with uninstallation feature, correct > dependency handling: if you start installing things with setuptools > there, dpkg cannot know anymore how to manage your system. Agreed. > That's why it is generally a very bad idea to install things which > are not managed by dpkg in /usr - be it python or something else > BTW. It is a much better practice to install from source into > /usr/local, or your $HOME, etc... Anywhere which is not /usr. easy_install can do that I think... I find it odd that easy_install doesn't have a) a list what you installed with easy_install b) uninstall in an otherwise excellent program. -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Learning Python now coming from Perl
Roy Smith <[EMAIL PROTECTED]> wrote: > In article <[EMAIL PROTECTED]>, > Nick Craig-Wood <[EMAIL PROTECTED]> wrote: > > > My favourite mistake when I made the transition was calling methods > > without parentheses. In perl it is common to call methods without > > parentheses - in python this does absolutely nothing! pychecker does > > warn about it though. > > > > perl -> $object->method > > python -> object.method() > > On the other hand, leaving out the parens returns the function itself, > which you can then call later. I've often used this to create data-driven > logic. I didn't say it wasn't useful, just that if you came from Perl like I did, it is an easy mistake to make ;-) > For example, I'm currently working on some code that marshals objects of > various types to a wire protocol. I've got something like: > > encoders = { > SM_INT: write_int, > SM_SHORT: write_short, > SM_FLOAT: write_float, > # and so on > } > > class AnyVal: > def __init__(self, type, value): >self.type = type >self.value = value > > def write_anyval(any): > encoders[any.type](any.value) > > The fact that functions are objects which can be assigned and stored in > containers makes this easy to do. OO lore says whenever you see a type field in an instance you've gone wrong - types should be represented by what sort of object you've got, not by a type field. Eg http://www.soberit.hut.fi/mmantyla/BadCodeSmellsTaxonomy.htm """The situation where switch statements or type codes are needed should be handled by creating subclasses. """ Here is my first iteration (untested) class AnyVal: def __init__(self, value): self.value = value def write(self): raise NotImplementedError() class IntVal(AnyVal): def write(self): # write_int code class ShortVal(AnyVal): def write(self): # write_short code class FloatVal(AnyVal): def write(self): # write_float code Then to write an AnyVal you just call any.write() The initialisation of the AnyVals then becomes from AnyVal(int_expression, SM_INT) to IntVal(int_expression) However, if the types of the expressions aren't known until run time, then use a dict of class types AnyValRegistry = { SM_INT: IntVal, SM_SHORT: ShortVal, SM_FLOAT: FloatVal, # and so on } And initialise AnyVal objects thus any = AnyValRegistry[type](value) This smells of code duplication though and a real potential for a mismatch between the AnyValRegistry and the actual classes. I'd probably generalise this by putting the type code in the class and use a __metaclass__ to autogenerate the AnyValRegistry dict which would then become an attribute of AnyClass Eg (slightly tested) SM_INT=1 SM_SHORT=2 SM_FLOAT=3 class AnyVal(object): TYPE = None registry = {} class __metaclass__(type): def __init__(cls, name, bases, dict): cls.registry[cls.TYPE] = cls def __init__(self, value): self.value = value @classmethod def new(cls, type_code, value): """Factory function to generate the correct subclass of AnyVal by type code""" return cls.registry[type_code](value) def write(self): raise NotImplementedError() class IntVal(AnyVal): TYPE = SM_INT def write(self): # write_int code print "int", self.value class ShortVal(AnyVal): TYPE = SM_SHORT def write(self): # write_short code print "short", self.value class FloatVal(AnyVal): TYPE = SM_FLOAT def write(self): # write_float code print "float", self.value You then make new objects with any = AnyVal.new(type_code, value) and write them with any.write() Anyone can add a subclass of AnyVal and have it added to the AnyVal.registry which is neat. >>> any = AnyVal.new(SM_SHORT, 1) >>> any <__main__.ShortVal object at 0xb7e3776c> >>> any.write() short 1 >>> any = AnyVal.new(SM_FLOAT, 1.8) >>> any <__main__.FloatVal object at 0xb7e37a6c> >>> any.write() float 1.8 You could also override __new__ so you could write AnyVal(type_code, value) to create the object of a new type. I personally don't think its is worth it - a factory function is nice and obvious and show exactly what is going on. -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Learning Python now coming from Perl
Bertilo Wennergren <[EMAIL PROTECTED]> wrote: > I'm planning to start learning Python now, using Python 3000. > I have no previous Python skills, but I now Perl pretty well. > I'm also well experienced with JavaScript. > > Any pointers and tips how I should go about getting into > Python? Read "Dive Into Python" while following along with your keyboard. ( http://www.diveintopython.org/ free online or paper edition from your favourite bookseller ) My favourite mistake when I made the transition was calling methods without parentheses. In perl it is common to call methods without parentheses - in python this does absolutely nothing! pychecker does warn about it though. perl -> $object->method python -> object.method() -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How can I do this (from Perl) in Python? (closures)
Duncan Booth <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: > > > I just came across http://www.perl.com/pub/a/2002/05/29/closure.html > > and wanted to try the "canonical example of closures" in Python. I > > came up with the following, but it fails: > > > > def make_counter(start_num): > > start = start_num > > def counter(): > > start += 1 > > return counter > > The other answers you got will work, but here's an alternative to > consider: you can convert the function into a generator and then just > move the variable inside. > > >>> def make_counter(start_num): > def counter(): > start = start_num > while 1: > yield start > start += 1 > return counter().next Interesting... You can also write it without nested functions / closures def counter(x): while 1: yield x x += 1 def make_counter(start_num): return counter(start_num).next I expect the machinery is similar between generators and closures, but with generators it is a lot more obvious exactly which version of which variable you are using! In fact you can always do what you can do with a closure with the above technique I think... def make_closure(*args, **kwargs): # initialisation to local vars def closure(): # stuff, being careful with nonlocal args & kwargs return result vs def closure(*args, **kwargs): # initialisation to local vars while 1: # normal stuff using args and kwargs yield result def make_closure(*args, **kwargs): return closure(*args, **kwargs).next I still prefer doing it explicitly with a class though ;-) -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How can I do this (from Perl) in Python? (closures)
[EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > I just came across http://www.perl.com/pub/a/2002/05/29/closure.html > and wanted to try the "canonical example of closures" in Python. I > came up with the following, but it fails: > > def make_counter(start_num): > start = start_num > def counter(): > start += 1 > return counter > > from_ten = make_counter(10) > from_three = make_counter(3) > > print from_ten() # 10 > print from_ten() # 11 > print from_three() # 3 > print from_ten() # 12 > print from_three() # 4 > > The error message is: "UnboundLocalError: local variable 'start' > referenced before assignment". The same thing happens if I omit start > and just use start_num directly. > > How can I do it in Python? With a class is the best way IMHO. class make_counter(object): def __init__(self, start_num): self.x = start_num def __call__(self): x = self.x self.x += 1 return x -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to instantiate in a lazy way?
Slaunger <[EMAIL PROTECTED]> wrote: > On 3 Dec., 11:30, Nick Craig-Wood <[EMAIL PROTECTED]> wrote: > > > ? ? ? ? ?cls = self.__class__ > > > ? ? ? ? ?if attr_name in cls.data_attr_names: > > > > self.data_attr_names should do instead of cls.data_attr_names unless > > you are overriding it in the instance (which you don't appear to be). > > Yeah, I know. I just like the cls notation for code readability > because it tells you that it is a class attribute, which is not > instance- dependent. > > That may be legacy from my Java past, where I used to do it that > way. I know perfectly well that self. would do it. i just find > that notation a little misleading I quite like it... It looks in the instance, then in the class which I find to be very elegant - you can set a default in the class and override it on a per object or per subclass basis. > > I think you want setattr() here - __dict__ is an implemetation detail > > - classes with __slots__ for instance don't have a __dict__. ?I'd > > probably do this > > Oh my, I did not know that. __slots__?? Something new I got to > understand. But you are right. thanks! > > > > > ? ? ? ? ? ? ? ? ? ?for k, v in zip(("I1", "Q1", "I2", "Q2"), > > bytes_to_data(buf)): > > ? ? ? ? ? ? ? ? ? ? ? ?setattr(self, k, v) > > ? ? ? ? ? ? ? ? ? ?return object.__getattr__(self, attr_name) > > > And perhaps even more readable (how I do it now, no need to call > __getattr__ for an attribute, whcih is already there): > ... > for attr, value in zip(cls.data_attr_names, > bytes_to_data(buf)): > setattr(self, attr, value) > > return getattr(self, attr_name) I wrote the object.__getattr__ call to stop recursion troubles. If you are sure you've set the attribute then plain getattr() is OK I guess... > In this process I have actaully managed to isolate all the > ...OnDemand stuff in an abstract PayloadOnDemand class > > I can now use this "decorator-like"/helper class to very easily > make an ...OnDemand variant of a class by just doing multiple > inheritance - no implementation: > > class PayloadBaconAndEggsOnDemand(PayloadOnDemand, PayloadBaconAndEggs): pass You've reinvented a Mixin class! http://en.wikipedia.org/wiki/Mixin It is a good technique. -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to instantiate in a lazy way?
Slaunger <[EMAIL PROTECTED]> wrote: > Just wanted to show the end result in its actual implementation! > > I ended up *not* making a decorator, as I already had a good idea > about how to do it > using __getattr__ > > class PayloadDualFrqIQOnDemand(PayloadDualFrqIQ): > """ > This class has the same interface as its parent, > but unlike its parent, it is instantiated without > its payload parsed up in its instance attributes > Q1, I1, Q2 and I2. Instead it stores a reference to > the file object in which the Payload data can be > read, the file position and > the version of the payload data. > > On accessing one of the data attributes, the actual > payload data are read from the file, and the reference to > the file object is unbound. > The constructor signature is therefore different from its > parent as it takes the file object, position and version > as arguments instead of the actual data. > """ > > @classmethod > def _unpack_from_file(cls, f, samples, ver): > bytes = samples * cls.bytes_per_sample > initial_pos = f.tell() > f.seek(initial_pos + bytes) #Skip over the payload > return cls(f, initial_pos, samples, ver) > > @classmethod > def unpack_from_ver3_file(cls, f, samples): > return cls._unpack_from_file(f, samples, ver=3) > > @classmethod > def unpack_from_ver4_file(cls, f, samples): > return cls._unpack_from_file(f, samples, ver=4) > > data_attr_names = frozenset(["Q1", "I1", "Q2", "I2"]) > > def __init__(self, a_file, a_file_position, samples, a_version): > """ > Returns an instance where the object knows where to > look for the payload but it will only be loaded on the > first attempt to read one of the data attributes > in a "normal" PayloadDualFrqIQ object. > """ > self.f = a_file > self.file_position = a_file_position > self.samples = samples > self.ver = a_version > > def __getattr__(self, attr_name): > """ > Checks if a request to read a non-existing data attribute > has an attribute corresponding to one of the data attributes > in a normal PayloadDualFrqIQ object. > > If true, the data attributes are created and bound to the > object using the file object instance, the file position > and the version. > > It is a prerequisite that the file object is still open. > The function leaves the file object at the file position > when it entered the method > > """ > cls = self.__class__ > if attr_name in cls.data_attr_names: self.data_attr_names should do instead of cls.data_attr_names unless you are overriding it in the instance (which you don't appear to be). > initial_pos = self.f.tell() > try: > bytes = self.samples * cls.bytes_per_sample > self.f.seek(self.file_position) > buf = self.f.read(bytes) > if self.ver == 3: > bytes_to_data = cls._v3_byte_str_to_data > elif self.ver == 4: > bytes_to_data = cls._v4_byte_str_to_data > else: > raise TermaNotImplemented, \ > "Support for ver. %d not implemented." % > self.ver > I1, Q1, I2, Q2 = bytes_to_data(buf) > self.__dict__["I1"] = I1 > self.__dict__["Q1"] = Q1 > self.__dict__["I2"] = I2 > self.__dict__["Q2"] = Q2 > return self.__dict__[attr_name] I think you want setattr() here - __dict__ is an implemetation detail - classes with __slots__ for instance don't have a __dict__. I'd probably do this for k, v in zip(("I1", "Q1", "I2", "Q2"), bytes_to_data(buf)): setattr(self, k, v) return object.__getattr__(self, attr_name) That then has duplicated a list of "I1", "Q1" etc which needs to be factored out. > finally: > # Restore file position > self.f.seek(initial_pos) > # Unbind lazy attributes > del self.f > del self.ver >
Re: Do more imported objects affect performance
On Tue, Dec 02, 2008 at 10:53:47PM -0500, Steve Holden wrote: > Pardon me for intruding, but timings here are entirely the wrong focus > for a Python newcomer. Given that imports are super-optimized (i.e. the > code in the module is only performed once) such a small difference in > timing is inconsequential, I would suggest. > > As long as "from module import *" is only ever used with modules > specifically designed to support it, the other forms can be used as > required. Sure, there is a timing difference between > > import module > ... > module.something() > > and > > from module import something > ... > something() > > but that's hardly the point. Learning to write sound Python is *much* > more important that learning to write fast Python, and often the two > coincide anyway. > > It was true when Kernighan and Plauger wrote it forty years ago and it's > true now: "First, make it work. Then, *if it doesn't work fast enough*, > make it work faster". You are 100% right of course Steve. I was just trying to answer the specific question "which is faster" question which probably isn't helpful for new Python programmers to focus on. PS I enjoyed your book :-) -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Confused about class relationships
what you have is a totally acceptable factory system. Not sure why you are using a generator, but that's another matter. I agree with the previous replies regarding inheritance... this is not a case for inheritance. You could, however, have Bar be a borg with the Bar factory built in as a class method to the Bar class itself if you want the whole job done by one component. Questions to lead to or away from that: will Engine every create things besides Bars? -- http://mail.python.org/mailman/listinfo/python-list
Re: HELP!...Google SketchUp needs a Python API
> Just remember thought that if you threat Python like a > hammer, suddenly everything will look like a bail. > don't you mean if you use Python like a pitchfork? -- http://mail.python.org/mailman/listinfo/python-list
Re: Do more imported objects affect performance
Steven D'Aprano <[EMAIL PROTECTED]> wrote: > On Tue, 02 Dec 2008 11:12:31 +0000, Nick Craig-Wood wrote: > > > I prefer the "from module import function". That means that if "module" > > doesn't supply "function" it raises an exception at compile time, not > > run time when you try to run "module.function". > > Wanna bet? > > > >>> def spam(): > ... from math import harmonic_series > ... return harmonic_series() > ... > >>> dis.dis(spam) >2 0 LOAD_CONST 1 (-1) >3 LOAD_CONST 2 (('harmonic_series',)) >6 IMPORT_NAME 0 (math) >9 IMPORT_FROM 1 (harmonic_series) > 12 STORE_FAST 0 (harmonic_series) > 15 POP_TOP > >3 16 LOAD_FAST0 (harmonic_series) > 19 CALL_FUNCTION0 > 22 RETURN_VALUE > >>> spam() > Traceback (most recent call last): >File "", line 1, in >File "", line 2, in spam > ImportError: cannot import name harmonic_series > > > The same thing happens if the from...import is at the top level of the > module, except that compilation is immediately followed by > execution. You are technically right I am sure. However the error happens when you import the module with the error in, not when you run stuff from it which is the major difference. $ echo -e "from os import sausage\n" > import_test.py $ python Python 2.5.2 (r252:60911, Sep 29 2008, 21:15:13) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> # This does produce an error >>> import import_test Traceback (most recent call last): File "", line 1, in File "import_test.py", line 1, in from os import sausage ImportError: cannot import name sausage >>> $ # This produces an error also $ python import_test.py Traceback (most recent call last): File "import_test.py", line 1, in from os import sausage ImportError: cannot import name sausage $ Unlike $ echo -e "import os\ndef f(): os.sausage\n" > import_test.py $ python Python 2.5.2 (r252:60911, Sep 29 2008, 21:15:13) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> # This doesn't produce an error >>> import import_test >>> # Until you actually call it >>> import_test.f() Traceback (most recent call last): File "", line 1, in File "import_test.py", line 2, in f def f(): os.sausage AttributeError: 'module' object has no attribute 'sausage' >>> $ # No error here either $ python import_test.py $ > > It then becomes very easy to see which functions you use from any > > given module too. > > If that's important to you. Personally, I find it more useful to know > where a function is defined. We can agree to differ there I'm sure ;-) -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to instantiate in a lazy way?
Slaunger <[EMAIL PROTECTED]> wrote: > On 2 Dec., 11:30, Nick Craig-Wood <[EMAIL PROTECTED]> wrote: > > > > > For 4 attributes I'd probably go with the __getattr__. > > > OK, I'll do that! > > > Or you could easily write your own decorator to cache the result... > > > > Eg http://code.activestate.com/recipes/363602/ > > Cool. I never realized I could write my own decorators! > I've so far only used them for > @classmethod, @staticmethod and stuff like that. > User defined decorators are nice and fun to do as well. > I just hope it will be understandable > in four years also... Actually that decorator does exactly what you want. It replaces the attribute with the actual data so after the first call there is no overhead. I'd use that I think! > > > ?With the property methology you do the value check on each get, which > > > ?does not look as "clean". The property methology is also a little less > > > ?arcane I guess for less experienced Python programmers to understand > > > ?when re-reading the code. > > > > Less magic is how I would put it. ?Magic is fun to write, but a pain > > to come back to. ?Over the years I find I try to avoid magic more and > > more in python. > > > Ah, I see. I hope you do not consider user defined decorators > "magic" then? ;-) Well the magic is contained in the decorator so it isn't splattered throughout your code! Decorators are part of the meta-programming toolbag along with metaclasses. Used sparingly they can be extrememly useful! > Again, thank you for your assistance, Nick! No problem! -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list