Re: ANN: GMPY 1.11rc1 is available
casevh wrote: > I'm pleased to annouce that a new version of GMPY is available. > GMPY is a wrapper for the MPIR or GMP multiple-precision > arithmetic library. GMPY 1.11rc1 is available for download from: [snip] > Future plans > > On releasing the GIL: I have compared releasing the GIL versus the > multiprocessing module and the multiprocessing module offers better > and more predictable performance for embarrassingly parallel tasks > than releasing the GIL. If there are requests, I can add a compile- > time option to enable threading support but it is unlikely to > become the default. You could have variant types of mpz etc which do release the GIL so you could mix and match accordingly. > On mutable integers: The performance advantages of mutable integers > appears to be 20% to 30% for some operations. I plan to add a new > mutable integer type in the next release of GMPY. If you want to > experiment with mutable integers now, GMPY can be compiled with > mutable version of the standard 'mpz' type. Please see the file > "mutable_mpz.txt" for more information. Mutable integers - whatever will they think of next ;-) Thanks for maintaining gmpy - it is an excellent bit of software! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Looking for a pure Python chart drawing module
John Nagle wrote: > I'm looking for something that can draw simple bar and pie charts > in Python. I'm trying to find a Python package, not a wrapper for > some C library, as this has to run on both Windows and Linux > and version clashes are a problem. > > Here's the list from the Python wiki at > "http://wiki.python.org/moin/NumericAndScientific/Plotting";. > Almost all the options are really wrappers for some other > package in C/C++. [snip] > So, for pure Python, Pychart is it. I'll have to try it and see if it still > works. I don't think anyone has mentioned reportlab... It can plot charts I think, though last time I used it I plotted stuff by hand as I wanted exact control over the layout. I'm not sure of the dependencies though so may not be suitable for your purposes. http://www.reportlab.org/rl_toolkit.html -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: VT100 in Python
Wolfgang Rohdewald wrote: > On Sunday 13 September 2009, Nadav Chernin wrote: > > I'm writing program that read data from some instrument trough > > RS232. This instrument send data in VT100 format. I need only to > > extract the text without all other characters that describe how to > > represent data on the screen. Is there some library in python for > > converting VT100 strings? > > that should be easy using regular expressions At a basic level parsing VT100 is quite easy, so you can get rid of the VT100 control. They start with ESC, have other characters in the middle then end with a letter (upper or lowercase), so a regexp will make short work of them. Something like r"\x1B[^A-Za-z]*[A-Za-z]" You might need to parse the VT100 stream as VT100 builds up a screen buffer though and the commands don't always come out in the order you might expect. I think twisted has VT100 emulator, but I couldn't find it in a brief search just now. You'll find various others (like this one) if you search some more http://svn.python.org/projects/python/branches/string_methods/Demo/cwilib/vt100.py -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: including constants
Nikola Skoric wrote: > I have a simple problem and I know how to solve it :-D, but I suspect that > there is a standard solution which is more elegant. > > So, here is my problem: I have django app versioned with svn and I > test my trunk on two different machines (one with mysql, another with > sqlite3). How do I remove database-definition constants from my > settings.py and put them in a separate file which wouldn't be > maintained by svn? (did I really hear somebody call that kind of file > "unversioned file" or did I make that up?) I suppose I could make a > trivial module with database info, import it and then assign my > constants with data from that module, but that looks so unelegant to > me. What's the standard way of doing this? Well settings.py is a normal python program so you can put any mechanisms you can think of in there. For example (from simple to complex) Make settings.py a symlink to the correct one for the machine. Make a subsettings.py which is a symlink to the correct one for the machine. Pass in some of the settings in the environment and read them with os.environ. Look up the hostname of the machine and import a secondary file of settings with that name. This has the advantage that you can check everything in. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: CPU usage while reading a named pipe
Miguel P wrote:
> On Sep 12, 2:54 pm, Ned Deily wrote:
> > In article
> > ,
> > Miguel P wrote:
> > > I've been working on parsing (tailing) a named pipe which is the
> > > syslog output of the traffic for a rather busy haproxy instance. It's
> > > a fair bit of traffic (upto 3k hits/s per server), but I am finding
> > > that simply tailing the file in python, without any processing, is
> > > taking up 15% of a CPU core. In contrast HAProxy takes 25% and syslogd
> > > takes 5% with the same load. `cat < /named.pipe` takes 0-2%
> >
> > > Am I just doing things horribly wrong or is this normal?
> >
> > > Here is my code:
> >
> > > from collections import deque
> > > import io, sys
> >
> > > WATCHED_PIPE = '/var/log/haproxy.pipe'
> >
> > > if __name__ == '__main__':
> > > try:
> > > log_pool = deque([],1)
> > > fd = io.open(WATCHED_PIPE)
> > > for line in fd:
> > > log_pool.append(line)
> > > except KeyboardInterrupt:
> > > sys.exit()
> >
> > > Deque appends are O(1) so that's not it. And I am using 2.6's io
> > > module because it's supposed to handle named pipes better. I have
> > > commented the deque appending line and it still takes about the same
> > > CPU.
> >
> > Be aware that the io module in Python 2.6 is written in Python and was
> > viewed as a prototype. In the current svn trunk, what will be Python
> > 2.7 has a much faster C implementation of the io module backported from
> > Python 3.1.
>
> Aha, I will test with trunk and see if the performance is better, if
> so I'll use 2.6 in production until 2.7 comes out. I will report back
> when I have made the tests.
Why don't you try just using the builtin open() with bufsize
parameter set big?
Something like this (tested with named pipes). Tweak BUFFERSIZE and
SLEEP_INTERVAL for maximum performance!
import time
BUFFERSIZE = 1024*1024
SLEEP_INTERVAL = 0.1
def tail(path):
fd = open(path)
buf = ""
while True:
buf += fd.read(BUFFERSIZE)
if buf:
lines = buf.splitlines(True)
for line in lines[:-1]:
yield line
buf = lines[-1]
if buf.endswith("\n"):
yield buf
buf = ""
else:
time.sleep(SLEEP_INTERVAL)
def main(path):
for line in tail(path):
print "%r:%r" % (len(line), line)
if __name__ == "__main__":
import sys
main(sys.argv[1])
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Application-global "switches"?
kj wrote: > I'm looking for the "best-practice" way to define application-global > read-only switches, settable from the command line. The best > example I can think of of such global switch is the built-in variable > __debug__. This variable is visible everywhere in a program, and > broadly affects its operation. This is what I've done in the past... Create a Config class in a module called config.py and instantiate it class Config: def __init__(self): self.debug = True # etc config = Config() Then where you want to use it you can then use from config import config if config.debug: # blah This has the advantage that you can define some methods on your config object (eg save). I don't know whether this is best practice but it works for me! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to create ones own lib
Horst Jäger wrote: > I would like to create my own lib "hotte.py" which I can import like > > import string,hotte > > . How do I do that? One of the nice things about python is that you don't need to do anything special to define a library. Just define your classes / functions in hotte.py then use them like import hotte hotte.MyClass() hotte.my_function() See here for the relevant bit of the tutorial http://docs.python.org/tutorial/modules.html -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Does python have the capability for driver development ?
Diez B. Roggisch wrote: > MalC0de schrieb: > > hello there, I've a question : > > I want to know does python have any capability for using Ring0 and > > kernel functions for driver and device development stuff . > > if there's such a feature it is very good, and if there something for > > this kind that you know please refer me to some reference and show me > > some snippet . > > No, it can't do such things. At least it isn't embedded in the kernel - > in theory that might be possible, but given the timing-constraints and > concurrency-requirements, it's not really feasible. You can write FUSE (file systems in userspace) drivers in python I believe. Not the same as running in ring0 but in most senses a kernel driver... -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Semaphore Techniques
John D Giotta wrote:
> I'm looking to run a process with a limit of 3 instances, but each
> execution is over a crontab interval. I've been investigating the
> threading module and using daemons to limit active thread objects, but
> I'm not very successful at grasping the documentation.
>
> Is it possible to do what I'm trying to do and if so anyone know of a
> useful example to get started?
If you want a simple, cross platform way of doing it, then bind each
process to a different local tcp port.
Make a list of 3 ports, and try binding to each port in turn. If you
can't find a port to bind to then there are already 3 instances
running.
Something like this
import socket
PORTS = range(1,10003)
lock_sock = None
def lock_process(_locks = []):
for port in PORTS:
sock = socket.socket()
try:
sock.bind(("localhost", port))
except socket.error, e:
sock = None
else:
_locks.append(sock)
break
else:
raise Exception("Too many instances of me running")
for i in range(5):
print "Trying",i+1
lock_process()
Which prints
Trying 1
Trying 2
Trying 3
Trying 4
Traceback (most recent call last):
File "", line 20, in
File "", line 16, in lock_process
Exception: Too many instances of me running
You could do the same thing with lock files also very easily...
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: bad certificate error
jakecjacobson wrote: > > > If I go to this server in my browser, I get a "This server tried to > > > identify itself with invalid information". Is there a way to > > > ignore this issue with Python? Can I setup a trust store and add > > > this server to the trust store? > > > > Invalid how? Self signed certificate? Domain mismatch? Expired > > certificate? > > For everyone that wants to discuss why we shouldn't do this, great but > I can't change the fact that I need to do this. I can't use http or > even get a correct cert at this time. This is a quick a dirty project > to demonstrate capability. I need something more than slide show > briefs. I wasn't making a value judgement - I was trying to help! If you can get a bit more information out of the browser as to why the certificate is invalid it may help your python code. Real certificates cost real money. Usually a correctly set up self-signed certificate is fine for dev stuff. I'm certainly too cheap to by real certificates for dev or internal stuff! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: bad certificate error
jakecjacobson wrote:
> I am getting the following error when doing a post to REST API,
>
> Enter PEM pass phrase:
> Traceback (most recent call last):
>File "./ices_catalog_feeder.py", line 193, in ?
> main(sys.argv[1])
>File "./ices_catalog_feeder.py", line 60, in main
> post2Catalog(catalog_host, catalog_port, catalog_path, os.path.join
> (input_dir, file), collection_name, key_file, cert_file)
>File "./ices_catalog_feeder.py", line 125, in post2Catalog
> connection.request('POST', path, parameters, head)
>File "/usr/lib/python2.4/httplib.py", line 810, in request
> self._send_request(method, url, body, headers)
>File "/usr/lib/python2.4/httplib.py", line 833, in _send_request
> self.endheaders()
>File "/usr/lib/python2.4/httplib.py", line 804, in endheaders
> self._send_output()
>File "/usr/lib/python2.4/httplib.py", line 685, in _send_output
> self.send(msg)
>File "/usr/lib/python2.4/httplib.py", line 652, in send
> self.connect()
>File "/usr/lib/python2.4/httplib.py", line 1079, in connect
> ssl = socket.ssl(sock, self.key_file, self.cert_file)
>File "/usr/lib/python2.4/socket.py", line 74, in ssl
> return _realssl(sock, keyfile, certfile)
> socket.sslerror: (1, 'error:14094412:SSL
> routines:SSL3_READ_BYTES:sslv3 alert bad certificate')
>
>
> My code where this error occurs is:
>
> head = {"Content-Type" : "application/x-www-form-urlencoded",
> "Accept" : "text/plain"}
> parameters = urlencode({"collection" : collection, "entryxml" : open
> (file,'r').read()})
> print "Sending the file to: " + host
>
> try:
> try:
> # Default port is 443.
> # key_file is the name of a PEM formatted file that contains
> your
> private key.
> # cert_file is a PEM formatted certificate chain file.
> connection = httplib.HTTPSConnection(host, int(port), key_file,
> cert_file)
> connection.request('POST', path, parameters, head)
> response = connection.getresponse()
> print response.status, response.reason
> except httplib.error, (value,message):
> print value + ':' + message
> finally:
> connection.close()
>
> I was wondering if this is due to the server having a invalid server
> cert?
I'd say judging from the traceback you messed up key_file or cert_file
somehow.
Try using the openssl binary on them (read the man page to see how!)
to check them out.
> If I go to this server in my browser, I get a "This server tried to
> identify itself with invalid information". Is there a way to
> ignore this issue with Python? Can I setup a trust store and add
> this server to the trust store?
Invalid how? Self signed certificate? Domain mismatch? Expired certificate?
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Looking for os.listdir() generator
Christian Heimes wrote: > Nick Craig-Wood wrote: > > Christian Heimes wrote: > >> I'm looking for a generator version of os.listdir() for Python 2.5 and > >> newer. I know somebody has worked on it because I've seen a generator > >> version in a posting on some list or blog a while ago. I can't find it > >> anymore. It seems my Google fu is lacking today. All I can find is a > >> very old version of xlistdir. A Cython based solution is appreciated but > >> please no ctypes version. > > > > I posted exactly that yesterday I think ;-) > > > > Note that this has a python part as well as a ctypes part because the > > version of ctypes I used doesn't support generators. I think the > > development version does though. > > Thanks Nick! > > ctypes? I'm sure you wanted to say Cython :) Er, yes! > If you don't mind I'm going to wrap it up into a nice package and > release it on PyPI. Yes thats fine with me! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Looking for os.listdir() generator
Christian Heimes wrote:
> I'm looking for a generator version of os.listdir() for Python 2.5 and
> newer. I know somebody has worked on it because I've seen a generator
> version in a posting on some list or blog a while ago. I can't find it
> anymore. It seems my Google fu is lacking today. All I can find is a
> very old version of xlistdir. A Cython based solution is appreciated but
> please no ctypes version.
I posted exactly that yesterday I think ;-)
Note that this has a python part as well as a ctypes part because the
version of ctypes I used doesn't support generators. I think the
development version does though.
Here is it
--setup.py--
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("directory", ["directory.pyx"])]
)
--directory.pyx--
# Cython interface for listdir
# python setup.py build_ext --inplace
import cython
cdef extern from "dirent.h":
struct dirent:
char d_name[0]
ctypedef struct DIR
DIR *opendir(char *name)
int closedir(DIR *dirp)
dirent *readdir(DIR *dirp)
cdef extern from "errno.h":
int errno
cdef extern from "string.h":
char *strerror(int errnum)
cdef class Directory:
"""Represents an open directory"""
cdef DIR *handle
def __init__(self, path):
self.handle = opendir(path)
if self.handle is NULL:
raise OSError(errno, "Failed to open directory: %s" %
strerror(errno))
def readdir(self):
"""Read the next name in the directory"""
cdef dirent *p
p = readdir(self.handle)
if p is NULL:
return None
return p.d_name
def close(self):
"""Close the directory"""
if self.handle is not NULL:
closedir(self.handle)
self.handle = NULL
def __dealloc__(self):
self.close()
--listdir.py--
from directory import Directory
def listdir(path):
"""
A generator to return the names of files in the directory passed in
"""
d = Directory(path)
while True:
name = d.readdir()
if not name:
break
if name not in (".", ".."):
yield name
d.close()
del d
if __name__ == "__main__":
import sys
paths = sys.argv[1:]
if not paths:
paths = ["."]
for path in paths:
print "*** %s ***" % path
for name in listdir(path):
print name
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Pep 342 (val = yield MyGenerator(foo)), synchronous os.system() that doesn't block gui event loops
Ville Vainio wrote:
> Has anyone implementing something like what the subject line
> indicates?
>
> The idea:
>
> To run functions that execute a series of system commands without
> blocking the ui, *and* without adding state machine logic.
>
> The syntax would be something like:
>
> def work():
>
>showstatus("building")
>r = yield runshell("make")
>showstatus("installing")
>r = yield runshell("make install")
>showstatus("Success")
>
> mygui.startwork(work)
> # returns immediately, runs work() gradually in the background.
>
> The catch is that showstatus() would need to be run in the mainloop,
> so running the whole thing in a thread is a no-go.
>
> I imagine runshell() would be implemented in terms of QProcess, or
> subprocess.Popen/os.system and a worker thread.
>
> Anyone done this already, or do I have to roll my own?
You might want to look at twisted, in particular
http://twistedmatrix.com/trac/wiki/DeferredGenerator
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: sqlite3 performance problems only in python
David Stanek wrote: > On Thu, Jul 23, 2009 at 9:02 AM, Stef Mientki wrote: > > > > btw, I don't know if it's of any importance, the SQL-statement I perform is > > select OPNAMEN.*, NAME, NAME_, SCORES.SCORE, PATIENT.* > > from OPNAMEN > > inner join POID_VLID on OPNAMEN.POID = POID_VLID.POID > > inner join VRAAGLST on VRAAGLST.VLID = POID_VLID.VLID > > inner join VLID_SSID on VRAAGLST.VLID = VLID_SSID.VLID > > inner join SUBSCHAAL_GEGEVENS on SUBSCHAAL_GEGEVENS.SSID = VLID_SSID.SSID > > inner join POID_SSID_SCID on ( OPNAMEN.POID = > > POID_SSID_SCID.POID ) and > > ( SUBSCHAAL_GEGEVENS.SSID = > > POID_SSID_SCID.SSID ) > > inner join SCORES on SCORES.SCID = > > POID_SSID_SCID.SCID > > inner join PID_POID on OPNAMEN.POID = PID_POID.POID > > inner join PATIENT on PATIENT.PID = PID_POID.PID > > where substr ( lower( NAME) , 1, 6) = 'cis20r' > > and lower ( NAME_ ) = 'fatigue' > > and TEST_COUNT in (3,4) > > and DATETIME > 39814.0 > > and SCORE < 30 > > Warning: I suck at SQL and hate it with a passion... > > By using lower() on the left side of the where expressions I believe > that you are table scanning. So it is not the size of the data > returned, but the size of the data that needs to be scanned. In all the databases I've used, the like operator has been case insensitive, so if that is the problem you could use NAME like '%cis20r%' -- not quite the same, but close! and NAME_ like 'fatigue' instead which might be quicker. Or not ;-) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: extract c/cpp include file with regular expression
tiefeng wu wrote: > I need to parse c/cpp source files, one requirement is to extract > included header file name. If you are serious about parsing C code then you'll want to investigate gcc-xml http://www.gccxml.org/HTML/Index.html This runs the gcc frontend over the code but instead of producing an object file, it produces a machine readable xml file describing the source. It is used by h2xml.py / xml2py.py to make ctypes header file automatically. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Shed Skin 0.2, an experimental (restricted) Python-to-C++ compiler
srepmub wrote: > please send any program that doesn't work with shedskin (it still is > experimental after all) to me, or create an issue at > shedskin.googlecode.com, and I will have a look at the problem. I divided and conquered the program as suggested and eventually I got it to compile and run correctly :-) I learnt that if you have lots of variables with indeterminate type then shedskin takes a very long time indeed before blowing up! I also learnt that shedskin doesn't support the idiom I'd been using for creating shallow copies, namely the Board.__new__(Board) below. shedskin compiles it ok, but the C++ won't compile complaning about not being able to find __init__ methods Producing these warnings *WARNING* rush_hour_solver_cut_down.py:71: class 'Vehicle' has no method '__new__' *WARNING* rush_hour_solver_cut_down.py:72: variable 'new' has no type *WARNING* rush_hour_solver_cut_down.py:236: variable 'new_vehicle' has no type And these compile errors rush_hour_solver_cut_down.cpp:94: error: ‘__new__’ is not a member of ‘__rush_hour_solver_cut_down__::Vehicle’ rush_hour_solver_cut_down.cpp:95: error: expected type-specifier before ‘;’ token rush_hour_solver_cut_down.cpp: In member function ‘void* __rush_hour_solver_cut_down__::Board::move(int, int)’: rush_hour_solver_cut_down.cpp:276: error: ‘void*’ is not a pointer-to-object type rush_hour_solver_cut_down.cpp:276: error: ‘void*’ is not a pointer-to-object type rush_hour_solver_cut_down.cpp:279: error: ‘void*’ is not a pointer-to-object type rush_hour_solver_cut_down.cpp:279: error: ‘void*’ is not a pointer-to-object type rush_hour_solver_cut_down.cpp:281: error: invalid conversion from ‘void*’ to ‘__rush_hour_solver_cut_down__::Vehicle*’ def copy(self): new = Board.__new__(Board) new.me_x = self.me_x new.me_y = self.me_y new.depth = self.depth new.parent = self new.best_child = None new.board = [self.board[i][:] for i in range(WIDTH)] new.rep = self.rep[:] new.vehicles = self.vehicles[:] return new I changed to using copy.copy which did work, but I couldn't name my copy methods "copy" otherwise I got this error from the C++ compile rush_hour_solver_cut_down.cpp: In member function '__rush_hour_solver_cut_down__::Vehicle* __rush_hour_solver_cut_down__::Vehicle::copy()': rush_hour_solver_cut_down.cpp:94: error: no matching function for call to '__rush_hour_solver_cut_down__::Vehicle::copy(__rush_hour_solver_cut_down__::Vehicle* const)' rush_hour_solver_cut_down.cpp:89: note: candidates are: __rush_hour_solver_cut_down__::Vehicle* __rush_hour_solver_cut_down__::Vehicle::copy() rush_hour_solver_cut_down.cpp: In member function '__rush_hour_solver_cut_down__::Board* __rush_hour_solver_cut_down__::Board::copy()': rush_hour_solver_cut_down.cpp:135: error: no matching function for call to '__rush_hour_solver_cut_down__::Board::copy(__rush_hour_solver_cut_down__::Board* const)' rush_hour_solver_cut_down.cpp:129: note: candidates are: __rush_hour_solver_cut_down__::Board* __rush_hour_solver_cut_down__::Board::copy() So I renamed them to pycopy, and they ended up looking like def pycopy(self): new = copy(self) new.parent = self new.best_child = None new.board = [self.board[i][:] for i in range(WIDTH)] new.rep = self.rep[:] new.vehicles = self.vehicles[:] return new After all that - some timing results! Python: 9.3 seconds Psyco:5.8 seconds ShedSkin: 1.0 seconds Impressive! I put the code http://www.craig-wood.com/nick/pub/rush_hour_solver_cut_down.py I left in the commented out bits of code I had to change. This is only part of the project (375 lines) - it solves Rush Hour boards. There is another part which I haven't attempted to compile yet which finds the most difficult possible boards using a combination of back tracking and a genetic algorithm. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Shed Skin 0.2, an experimental (restricted) Python-to-C++ compiler
Mark Dufour wrote: > I have just released version 0.2 of Shed Skin, an experimental > (restricted) Python-to-C++ compiler (http://shedskin.googlecode.com). > It comes with 7 new example programs (for a total of 40 example > programs, at over 12,000 lines) and several important improvements/bug > fixes. See http://code.google.com/p/shedskin/wiki/ReleaseNotes for the > full changelog. Cool! I tried it on a program I wrote to solve a puzzle (Rush Hour). (Actually it solves the meta-puzzle - trying to make the hardest possible Rush Hour puzzle.) After a bit of twiddling (remove psyco and profiling) I got it to start compiling, but unfortunately it compiled for about an hour (in iterative type analysis..) used up 600 MB of RAM printing an '*' every 10 minutes or so then gave an error message and gave up. Unfortunately I shut the window by accident, but the error message was something about not being able to resolve types I think with a list of 20 or so unresolved types. Can you give a hint as to how I debug this? I presume my program has some instances of non static types which is causing the problem, but it is going to be a very long debugging session if it takes me an hour each cycle ;-) The program is about 700 lines of python (excluding comments). Thanks Nick -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: ctype performance benchmark
Wai Yip wrote:
> I started with ctypes because it is the battery included with the
> Python standard library. My code is very fluid and I'm looking for
> easy opportunity to optimize it. One example is to find the longest
> common prefix among two strings. Right now I am comparing it character
> by character with pure Python. It seems like an ideal low hanging
> fruit. With ctype I can rewriting a few lines of Python into a few
> lines of C. All the tools are available and no third party library is
> needed.
Yes ctypes is very easy for this and I've used it like this in the
past too. It makes interfacing C code very easy.
You might consider just writing a plain bit of C code. If you make a
setup.py too (not hard) then it is very easy to distribute and use on
other platforms.
The Python C API is very easy to use - documentation is extensive.
You need to be very careful with references and the error checking is
boring and tedious!
> It turned out the performance fail to match pure Python. This function
> is called million of times in an inner loop. The overhead overwhelm
> any performance gain with C. Eventually I have found success grouping
> the data into larger chunk for each API call. This is what I
> originally planned to do anyway. I am only sharing my experience here
> that doing fine grained ctype function call has its limitation.
Interesting but not too unexpected - that translation of types is
non-trivial.
> I have looked into Pyrex before and I was quite confused by the fusion
> of C and Python langauage. Perhaps it is time for me to give it a
> second look. I just heard of Cython now and it also look interesting.
> I think it is helpful for both project to articulate more clearly what
> they are, how they achieve the performance gain and to give more
> guidance on good use cases. On the other hand, perhaps it is just me
> are confused because I don't know enough of the Python internal.
Pyrex evolved/was forked into Cython.
Cython allows you to write a subset of python code that is compiled
into C. You can also optionally provide some type annotations to get
it to produce better C code at the cost of making your code look less
like python.
I find Cython alternatively brilliant and frustrating! Sometimes it
works really well, and sometimes I spend hours working out why my
program doesn't compile, or to find out the exact syntax I need to
interface with such and such a library.
I did a C library interfacing project with both ctypes and cython
recently as a comparison. It came out to be about the same number of
lines of code for both. I decided to run with the ctypes version as
it was part python.
Here is a short Cython example which interfaces with opendir /
closedir / readdir which python doesn't do natively. Note the way you
interface with C structures (the real ones are used in the C code -
the annotations just tell cython how to use them). Note also that
Directory is a class defined in C which I don't think you can't do
with ctypes without a wrapper class.
This compiles into 1320 lines of C, which in turn compile into 11380
bytes of shared object (when stripped).
import cython
cdef extern from "dirent.h":
struct dirent:
char d_name[0]
ctypedef struct DIR
DIR *opendir(char *name)
int closedir(DIR *dirp)
dirent *readdir(DIR *dirp)
cdef extern from "errno.h":
int errno
cdef extern from "string.h":
char *strerror(int errnum)
cdef class Directory:
"""Represents an open directory"""
cdef DIR *handle
def __init__(self, path):
self.handle = opendir(path)
if self.handle is NULL:
raise OSError(errno, "Failed to open directory: %s" %
strerror(errno))
def readdir(self):
"""Read the next name in the directory"""
cdef dirent *p
p = readdir(self.handle)
if p is NULL:
return None
return p.d_name
def close(self):
"""Close the directory"""
if self.handle is not NULL:
closedir(self.handle)
self.handle = NULL
def __dealloc__(self):
self.close()
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: This is a mess...
Nick wrote:
> this is the new oop version, its pretty messy currently, and i do
> understand it is a simple routine, but i'm using it as an exercise to
> learn oop python...
>
> first the (current) traceback:
> [:~/python]$ python oop_covariance.py b2ar_all_test b2ar_all_test
>
>
> Traceback (most recent call last):
>File "oop_covariance.py", line 24, in
> cov = set1.covariance(set2, Eigen_vect.dot)
>File "/home/nleioatts/python/Eigen.py", line 66, in covariance
> print self.vectors[i][i]
> AttributeError: Eigen_vect instance has no attribute '__getitem__'
You are trying to use your Eigen_vect class as an array. In order for
an object to act as an array it needs a __getitem__ method.
> class Eigen_vect:
> def __init__(self, e_val, e_vect):
> self.e_val = e_val
> self.e_vect = e_vect
> def length(self):
> return len(self.e_vect)
>
> def dot(self, other):
> d = 0.0
> if other.length() != self.length():
> raise ValueError, "Eigen Vectors not same Length"
> for k in range(self.length()):
> # print "HI NICK", self.e_vect[k], other.e_vect[k]
> d += float(self.e_vect[k]) * float(other.e_vect[k])
> return d
Either add a __getitem__ method like this
def __getitem__(self, n):
return self.e_vect[n]
Or you might want to subclass list if you are going to do a lot of
list like operations on Eigen_vects, eg
class Eigen_vect(list):
def __init__(self, e_val, e_vect):
self.e_val = e_val
self[:] = e_vect
def dot(self, other):
d = 0.0
if len(other) != len(self):
raise ValueError("Eigen Vectors not same Length")
for a, b in zip(self, other):
d += float(a) * float(b)
return d
Notice that now these things are lists, you can use len, zip etc on
them which makes the code much neater.
e = Eigen_vect(3, range(10))
f = Eigen_vect(4, range(1,11))
print e
print f
print e[2]
print e.dot(f)
Which prints
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
2
330.0
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: using timers to force an execution time
superpollo wrote:
> Diez B. Roggisch wrote:
> > superpollo wrote:
> >>am i wrong?
> >
> >
> > No, you are right, for threads that is. You can try & trick around with the
> > trace-functionality of python, and some ctypes-based
> > system-thread-module-tricks that are, as mentioned before, black-magic & a
> > spinning gatling gun pointed to your very own lower extremities.
> >
>
> OUCH.
>
> > The only reliable way of terminating an asynchronous computation is to use
> > processes, since python2.6 that's rather convenient with the
> > multiprocessing-module. However, that imposes some limits to what you can
> > do.
>
> fair enough.
Or if you are on unix you can use signals...
It is probably just about acceptable to raise an exception in a signal
handler like this code does.
import signal, os, time
class TimeOut(Exception):
"""Thrown on alarm"""
pass
def sig_alarm(signum, frame):
raise TimeOut()
def time_out(t, fn, *args, **kwargs):
"""Calls fn with the args and kwargs returning its result or raising a
TimeOut exception if it doesn't complete within t seconds"""
# Turn alarm off and read old value
old_alarm = signal.alarm(0)
# Install new handler remembering old
old_handler = signal.signal(signal.SIGALRM, sig_alarm)
# Set the timer going
signal.alarm(t)
try:
rc = fn(*args, **kwargs)
finally:
# Restore the old handler
signal.signal(signal.SIGALRM, old_handler)
signal.alarm(0)
def test():
for repeat in range(10):
print time.time()
time.sleep(0.66)
if __name__ == "__main__":
try:
time_out(3, test)
except TimeOut:
print "timed out"
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Catching control-C
Steven D'Aprano wrote: > On Mon, 06 Jul 2009 15:02:26 -0700, Michael Mossey wrote: > > > On Jul 6, 2:47 pm, Philip Semanchuk wrote: > >> On Jul 6, 2009, at 5:37 PM, Michael Mossey wrote: > >> > >> > What is required in a python program to make sure it catches a > >> > control- > >> > c on the command-line? Do some i/o? The OS here is Linux. > >> > >> You can use a try/except to catch a KeyboardInterrupt exception, or you > >> can trap it using the signal > >> module:http://docs.python.org/library/signal.html > >> > >> You want to trap SIGINT. > >> > >> HTH > >> Philip > > > > Thanks to both of you. However, my question is also about whether I need > > to be doing i/o or some similar operation for my program to notice in > > any shape or form that Control-C has been pressed. In the past, I've > > written Python programs that go about their business ignoring Ctrl-C. > > I bet that somewhere in your code you have something like: > > > for x in really_big_list: > try: > long_running_process(x) > except: > continue > > > If that's what you're doing, stop! The correct way is: > > > for x in really_big_list: > try: > long_running_process(x) > except Exception: > # Let KeyboardInterrupt and SystemExit through. > continue Note that it is a relatively recent change (in python 2.5) which made KeyboardInterrupt not a child of Exception n...@dogger:~$ python2.4 Python 2.4.6 (#2, Feb 17 2009, 20:01:48) [GCC 4.3.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. Loaded customisations from '/home/ncw/.pystartup' >>> isinstance(KeyboardInterrupt(), Exception) True >>> n...@dogger:~$ python2.5 Python 2.5.4 (r254:67916, Feb 17 2009, 20:16:45) [GCC 4.3.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. Loaded customisations from '/home/ncw/.pystartup' >>> isinstance(KeyboardInterrupt(), Exception) False >>> > for x in really_big_list: > try: > long_running_process(x) > except (KeyboardInterrupt, SystemExit): > print "User requested exit... shutting down now" > cleanup() > raise > except Exception: > continue That is the backwards compatible way -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Code that ought to run fast, but can't due to Python limitations.
John Nagle wrote:
> As an example of code that really needs to run fast, but is
> speed-limited by Python's limitations, see "tokenizer.py" in
>
> http://code.google.com/p/html5lib/
>
> This is a parser for HTML 5, a piece of code that will be needed
> in many places and will process large amounts of data. It's written
> entirely in Python. Take a look at how much work has to be performed
> per character.
>
> This is a good test for Python implementation bottlenecks. Run
> that tokenizer on HTML, and see where the time goes.
>
> ("It should be written in C" is not an acceptable answer.)
You could compile it with Cython though. lxml took this route...
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: how to spawn a process under different user
Gabriel Genellina wrote: > En Thu, 02 Jul 2009 19:27:05 -0300, Tim Harig > escribió: > > On 2009-07-02, sanket wrote: > >>> sanket wrote: > >>> > I am trying to use python's subprocess module to launch a process. > >>> > but in order to do that I have to change the user. > >> I am using python 2.4 on centos. > > > > I have never done this in python; but, using the normal system calls in C > > the process is basically: > > > > 1. fork() a new process > > 2. the child process changes its user id with setreuid() and > > possibly its group id with setregid() > > 3. then the child exec()s new process which replaces itself > > > > All of the necessary functions should be under the os module on POSIX > > operating systems. > > How to do that using the subprocess module: write a function for item (2) > above and pass it as the preexec_fn argument to Popen. preexec_fn is > executed after fork() and before exec() If you are forking 100s of processes a second you want to use the method above, however I think it is easier to use su (assuming you start off as root), so instead of passing ['mycommand', 'my_arg1', 'my_arg2'] to Popen, pass ['su', '-', 'username', '-c', 'mycommand my_arg1 my_arg2'] There is some opportunity for quoting problems there, but it is easy! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: C-extension 2 times slower than exe
Rolf Wester wrote: > Hello, > > thank you all very much for your replies. > > I tried to simplify things and make the two versions as comparable as > possible. I put the C++ part of the program into a shared object > libff.so. For the exe the main function is linked against this shared > object. For the python stuff I made an interface consiting of only one > function call_solver with the same code that has the main function used > for the exe. Then I created a wrapper for this interface using swig and > linked interface.o, ff_warp.o and libff.so into _ff.so. The Python code > just imports _ff and calls call_solver wich creates an object of the > class Solver and calls its member solve (the main function of the exe > does the same). > > I included some code for timing into the C++-code. > > #include > > //beginning of solve > clock_t t0 = clock(); > ... > clock_t t1 = clock(); > //end of solve > cout << "time used = " << (t1-t0)/CLOCKS_PER_SEC << endl; > > I'm using gcc4.5 (latest snapshot) and Python2.6 under Suse 10.3. The > sources are compiled using the flags -fPIC -O3. > > Timing: > > 1) time python ff.py > time used = 3.74 > real0m3.234s > user0m3.712s > sys 0m0.192s Those times look odd because the user time is > than the real time. User time is number of CPU seconds used. Real time is wallclock time. That must mean a) your program is threading b) there is something up with timing on your computer Looks odd but exactly what it means I don't know! > 2) time ff > time used = 2.19 > real0m3.170s > user0m2.088s > sys 0m0.168s -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Best way to enumerate classes in a module
Carl Banks wrote: > On Jun 23, 10:02 pm, Дамјан Георгиевски wrote: > > I need to programmaticaly enumerate all the classes in a given module. > > Currently I'm using dir(module) but the Notice on the documentation page > > [1] says "dir() is supplied primarily as a convenience for use at an > > interactive prompt" so that kind of scares me. > > > > Is there a better approach? > > > > If there is, how do I get all the classes of the current module? > > You can use module.__dict__.values() (or .itervalues()) to retrieve > the contents of the module (and of course .keys() if you want names). > If you want to check the same module that the code appears in, use > globals() instead of module.__dict__. > > Something makes me think that module.__dict__ was only added to Python > fairly recently, but I'm not sure. It exists in python2.1 - I don't have an older python to check at the moment. > A word of warning (although I would guess you are already aware of > these issues, but for other readers): this method can't tell the > difference between a class defined in the module and a class imported > into it. > > Finally, despite the warning, I think you are ok to use dir() for that > purpose. It's not likely to change. Good advice... And as a double check >>> import sys >>> set(sys.__dict__.keys()) == set(dir(sys)) True >>> import os >>> set(os.__dict__.keys()) == set(dir(os)) True -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: A superclass using a child classes' methods
Kurt Schwehr wrote: > Thanks for the excellent response setting me straight. Now to figure > out what dumb thing I did to make my code not work... If you have trouble then post the code (or a cut down version demonstrating the code) - it always helps to have real code with real error messages when tracking down problems. A lot of people (like me) will enjoy the puzzle of looking through your code and finding out where it went wrong. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Convert hash to struct
D'Arcy J.M. Cain wrote:
> On Fri, 19 Jun 2009 13:17:24 -0500
> Amita Ekbote wrote:
> > I am retrieving values from a database in the form of a dictionary so
> > I can access the values as d['column'] and I was wondering if there is
> > a way to convert the hash to a struct like format so i can just say
> > d.column. Makes it easier to read and understand.
>
> Are there enough clues here?
>
> class MyDict(dict):
> def __getattribute__(self, name):
> return dict.__getattribute__(self, name)
>
> def __getattr__(self, name):
> return self.get(name, 42)
>
> x = MyDict({'a': 1, 'b': 2, 'values': 3})
That is my preferred solution - subclass dict rather than make a new
type... I use this a lot for returning results from databases.
Here is a more fleshed out version. Changing KeyError to
AttributeError is necessary if you want the object to pickle.
class MyDict(dict):
"""
A dictionary with attribute access also.
If a builtin dictionary method collides with a member of the
dictionary, the member function will win.
"""
def __getattr__(self, name):
try:
return super(db_dict, self).__getitem__(name)
except KeyError:
raise AttributeError("%r object has no attribute %r" %
(self.__class__.__name__, name))
def __setattr__(self, name, value):
return super(db_dict, self).__setitem__(name, value)
def __delattr__(self, name):
try:
return super(db_dict, self).__delitem__(name)
except KeyError:
raise AttributeError("%r object has no attribute %r" %
(self.__class__.__name__, name))
def copy(self):
return MyDict(self)
> print x.a
> print x.z
> print x.values
>
> Big question - what should the last line display? If you expect "3"
> and not "" then
> you need to reconsider the above implementation. Thinking about the
> question may change your opinion about this being a good idea after
> all.
Indeed!
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Allocating memory to pass back via ctypes callback function
Scott wrote:
> I think I found the answer to my own question. Can anyone spot any
> issues with the following solution? The application I'm writing will
> be hitting these callbacks pretty heavily so I'm nervous about mucking
> up the memory management and creating one of those bugs that passes
> undetected through testing but nails you in production.
>
> def my_callback(p_cstring):
> answer = 'foobar'
>
> address = VENDOR_malloc(len(answer)+1)
>
> cstring = c_char_p.from_address( address )
I think this allocates the pointer (the c_char_p) in the malloced
block, not the actual data...
> cstring.value = answer
And this overwrites the pointer
> p_cstring.contents = cstring
> return
If you try this, it gives all sorts of rubbish data / segfaults
memmove(address, address+1, 1)
Here is how I'd do it
from ctypes import *
from ctypes.util import find_library
c_lib = CDLL(find_library("c"))
malloc = c_lib.malloc
malloc.argtypes = [c_long]
malloc.restype = c_void_p
answer = 'foobar\0'
address = malloc(len(answer))
print address
cstring = c_char_p()
print addressof(cstring)
cstring.value = address
memmove(address, answer, len(answer))
print cstring.value
memmove(address, address+1, 1)
print cstring.value
Which prints
159611736
3084544552
foobar
ooobar
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: ctypes list library
luca72 wrote: > There is a command for ctypes that help me to know the entry points > inside a library. I don't know.. However nm on the library works quite well on the command line $ nm --defined-only -D /usr/lib/libdl.so A GLIBC_2.0 A GLIBC_2.1 A GLIBC_2.3.3 A GLIBC_2.3.4 A GLIBC_PRIVATE 304c B _dlfcn_hook 13b0 T dladdr 1400 T dladdr1 0ca0 T dlclose 1170 T dlerror 1490 T dlinfo 1760 T dlmopen 0ae0 T dlopen 18d0 T dlopen 0cf0 T dlsym 0dd0 W dlvsym nm from the mingw distribution works on Windows too IIRC. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Packing a ctypes struct containing bitfields.
Karthik wrote:
> Hello Everybody,
>
> I'm trying to create a packed structure in ctypes (with one 64-bit
> element that is bitfielded to 48 bits),
> unsuccessfully:
>
> ===
> from ctypes import *
>
> class foo (Structure):
> _pack_ = 1
> _fields_ = [
> ("bar",c_ulonglong, 48),
> ]
>
> print("sizeof(foo) = %d" % sizeof(foo))
> ===
> I'm expecting that this should print 6 - however, on my box, it prints
> 8.
>
> The following piece of C code, when compiled and run, prints 6, which
> is correct.
> ===
> struct foo {
> unsigned long long bar: 48;
> };
>
> printf("sizeof(foo) = %d", sizeof(foo));
> ===
>
> So... what am I doing wrong?
I compiled and ran the above with gcc on my linux box - it prints 8
unless I add __attribute__((__packed__)) to the struct.
I'm not sure that using bitfields like that is a portable at all
between compilers let alone architectures.
I'd probably do
from ctypes import *
class foo (Structure):
_pack_ = 1
_fields_ = [
("bar0",c_uint32),
("bar1",c_uint16),
]
def set_bar(self, bar):
self.bar0 = bar & 0x
self.bar1 = (bar >> 32) & 0x
def get_bar(self):
return (self.bar1 << 32) + self.bar0
bar = property(get_bar, set_bar)
print "sizeof(foo) = %d" % sizeof(foo)
f = foo()
print f.bar
f.bar = 123456789012345
print f.bar
Which prints
sizeof(foo) = 6
0
123456789012345
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: UDP queue size
Martin P. Hellwig wrote: > Scott David Daniels wrote: > > æ¾å°èªå·±çä¸ç天 wrote: > >> I got a problem about UDP. > >> > >> How do I get the UDP buffer size? > >> > >> When the server had some delay in handling incoming UDP, it will lost > >> some package. I wonder it's because the system buffer size, is there any > >> ways to find the exactly size of the buffer? > > > > UDP is defined as "best effort then give up," so _everything_ that > > handles a UDP packet is allowed to drop it; that means switchers, > > routers, ..., anything along the path to the target application. > > So, you cannot expect to do any better than a conservative guess, > > perhaps augmented by dynamic scaling. > > I would like to add, that you are most likely to lose packages because > they are to big, I can not remember for sure if UDP had a fixed limit > above what the MTU does, but you might want to look at that > direction too. UDP messages can be up to 64k. However they will be fragmented to the MTU which is 1500 bytes for most ethernet traffic. The means that the chance of losing a bigger UDP message is much higher because you only need to lose one of the fragments to lose the message. Most people who use UDP try to keep the total message size below 1500 bytes (1480 data without the header IIRC) for that reason. wireshark/tcpdump will quickly show you whether you are fragmenting your UDP messages. Another thing to bear in mind with UDP is that you can easily exceed the packets / second that switches / routers can bear if you send lots of small messages. Switches / routers will start dumping packets if you do that since. Some switches just crash in my experience too ;-) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: UDP queue size
§ä´m¦Ã¤vª�...@¤ù¤à wrote:
> I got a problem about UDP.
>
> How do I get the UDP buffer size?
>
> When the server had some delay in handling incoming UDP, it will lost
> some package. I wonder it's because the system buffer size, is there any
> ways to find the exactly size of the buffer?
>
> ex:
>
> client.py
> import socket
>
> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> for i in xrange(1000):
> s.sendto('xxx', ('192.168.1.135',1))
>
>
>
> server.py: in ip (192.168.1.135)
> import socket
> import time
> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> s.bind(('',1))
> time.sleep(10)
>
> # here will only recv 255 package, others are lost...
> for i in xrange(1000):
> msg, addr = s.recvfrom(500)
> print i
I think you want setsockopt...
>>> import socket
>>> import time
>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> s.bind(('',1))
>>> s.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)
112640
>>> s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1048576)
>>> s.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)
262142
>>>
I ran the above on linux and I expect the limit 262144 is settable in
/proc/sys/net somewhere.
No idea whether the above works on windows!
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: waling a directory with very many files
Nick Craig-Wood wrote:
> Jean-Paul Calderone wrote:
> > On Mon, 15 Jun 2009 09:29:33 -0500, Nick Craig-Wood
> > wrote:
> > >Hrvoje Niksic wrote:
> > >> Nick Craig-Wood writes:
> > >>
> > >> > Here is a ctypes generator listdir for unix-like OSes.
> > >>
> > >> ctypes code scares me with its duplication of the contents of system
> > >> headers. I understand its use as a proof of concept, or for hacks one
> > >> needs right now, but can anyone seriously propose using this kind of
> > >> code in a Python program? For example, this seems much more
> > >> "Linux-only", or possibly even "32-bit-Linux-only", than
> > >> "unix-like":
> > >
> > >It was a proof of concept certainly..
Just in case anyone is interested here is an implementation using cython.
Compile with python setup.py build_ext --inplace
And run listdir.py
This would have been much easier if cython supported yield, but
unfortunately it doesn't (yet - I think it is in the works).
This really should work on any platform!
--setup.py--
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("directory", ["directory.pyx"])]
)
--directory.pyx--
# Cython interface for listdir
# python setup.py build_ext --inplace
import cython
cdef extern from "dirent.h":
struct dirent:
char d_name[0]
struct dir_handle:
pass
ctypedef dir_handle DIR "DIR"
DIR *opendir(char *name)
int closedir(DIR *dirp)
dirent *readdir(DIR *dirp)
cdef class Directory:
"""Represents an open directory"""
cdef DIR *handle
def __init__(self, path):
self.handle = opendir(path)
def readdir(self):
cdef dirent *p
p = readdir(self.handle)
if p is NULL:
return None
return p.d_name
def close(self):
closedir(self.handle)
--listdir.py--
from directory import Directory
def listdir(path):
"""
A generator to return the names of files in the directory passed
in
"""
d = Directory(".")
while True:
name = d.readdir()
if not name:
break
if name not in (".", ".."):
yield name
d.close()
if __name__ == "__main__":
for name in listdir("."):
print name
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: waling a directory with very many files
Jean-Paul Calderone wrote: > On Mon, 15 Jun 2009 09:29:33 -0500, Nick Craig-Wood > wrote: > >Hrvoje Niksic wrote: > >> Nick Craig-Wood writes: > >> > >> > Here is a ctypes generator listdir for unix-like OSes. > >> > >> ctypes code scares me with its duplication of the contents of system > >> headers. I understand its use as a proof of concept, or for hacks one > >> needs right now, but can anyone seriously propose using this kind of > >> code in a Python program? For example, this seems much more > >> "Linux-only", or possibly even "32-bit-Linux-only", than > >> "unix-like": > > > >It was a proof of concept certainly.. > > > >It can be done properly with gccxml though which converts structures > >into ctypes definitions. > > > >That said the dirent struct is specified by POSIX so if you get the > >correct types for all the individual members then it should be correct > >everywhere. Maybe ;-) > > The problem is that POSIX specifies the fields with types like off_t and > ino_t. Since ctypes doesn't know anything about these types, application > code has to specify their size and other attributes. As these vary from > platform to platform, you can't get it correct without asking a real C > compiler. These types could be part of ctypes. After all ctypes knows how big a long is on all platforms, and it knows that a uint32_t is the same on all platforms, it could conceivably know how big an off_t or an ino_t is too. > In other words, POSIX talks about APIs and ctypes deals with ABIs. > > http://pypi.python.org/pypi/ctypes_configure/0.1 helps with the problem, > and is a bit more accessible than gccxml. I haven't seen that before - looks interesting. > It is basically correct to say that using ctypes without using something > like gccxml or ctypes_configure will give you non-portable code. Well it depends on if the API is specified in types that ctypes understands. Eg, short, int, long, int32_t, uint64_t etc. A lot of interfaces are specified exactly like that and work just fine with ctypes in a portable way. I agree with you that struct dirent probably isn't one of those though! I think it would be relatively easy to implent the code I demonstrated in a portable way though... I'd do it by defining dirent as a block of memory and then for the first run, find a known filename in the block, establishing the offset of the name field since that is all we are interested in for the OPs problem. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: waling a directory with very many files
Hrvoje Niksic wrote: > Nick Craig-Wood writes: > > > Here is a ctypes generator listdir for unix-like OSes. > > ctypes code scares me with its duplication of the contents of system > headers. I understand its use as a proof of concept, or for hacks one > needs right now, but can anyone seriously propose using this kind of > code in a Python program? For example, this seems much more > "Linux-only", or possibly even "32-bit-Linux-only", than > "unix-like": It was a proof of concept certainly.. It can be done properly with gccxml though which converts structures into ctypes definitions. That said the dirent struct is specified by POSIX so if you get the correct types for all the individual members then it should be correct everywhere. Maybe ;-) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: waling a directory with very many files
tom wrote:
> On Jun 14, 1:35 pm, Tim Golden wrote:
> >
> > If you're on Windows, you can use the win32file.FindFilesIterator
> > function from the pywin32 package. (Which wraps the Win32 API
> > FindFirstFile / FindNextFile pattern).
>
> thanks, tim.
>
> however, i'm not using windows. freebsd and os x.
Here is a ctypes generator listdir for unix-like OSes. I tested it
under linux.
#!/usr/bin/python
"""
An equivalent os.listdir but as a generator using ctypes
"""
from ctypes import CDLL, c_char_p, c_int, c_long, c_ushort, c_byte, c_char,
Structure, POINTER
from ctypes.util import find_library
class c_dir(Structure):
"""Opaque type for directory entries, corresponds to struct DIR"""
c_dir_p = POINTER(c_dir)
class c_dirent(Structure):
"""Directory entry"""
# FIXME not sure these are the exactly correct types!
_fields_ = (
('d_ino', c_long),# inode number
('d_off', c_long),# offset to the next dirent
('d_reclen', c_ushort), # length of this record
('d_type', c_byte), # type of file; not supported by all file
system types
('d_name', c_char * 4096) # filename
)
c_dirent_p = POINTER(c_dirent)
c_lib = CDLL(find_library("c"))
opendir = c_lib.opendir
opendir.argtypes = [c_char_p]
opendir.restype = c_dir_p
# FIXME Should probably use readdir_r here
readdir = c_lib.readdir
readdir.argtypes = [c_dir_p]
readdir.restype = c_dirent_p
closedir = c_lib.closedir
closedir.argtypes = [c_dir_p]
closedir.restype = c_int
def listdir(path):
"""
A generator to return the names of files in the directory passed in
"""
dir_p = opendir(".")
while True:
p = readdir(dir_p)
if not p:
break
name = p.contents.d_name
if name not in (".", ".."):
yield name
closedir(dir_p)
if __name__ == "__main__":
for name in listdir("."):
print name
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Perl's @foo[3,7,1,-1] ?
kj wrote: > > > Switching from Perl here, and having a hard time letting go... > > Suppose I have an "array" foo, and that I'm interested in the 4th, 8th, > second, and last element in that array. In Perl I could write: > >my @wanted = @foo[3, 7, 1, -1]; > > I was a bit surprised when I got this in Python: > > >>> wanted = foo[3, 7, 1, -1] > Traceback (most recent call last): >File "", line 1, in > TypeError: list indices must be integers You've just tried to index a list with a tuple... If foo was a dictionary then this might make sense. > Granted, Perl's syntax is often obscure and hard-to-read, but in > this particular case I find it quite transparent and unproblematic, > and the fictional "pythonized" form above even more so. > > The best I've been able to come up with in Python are the somewhat > Perl-like-in-its-obscurity: > > >>> wanted = map(foo.__getitem__, (3, 7, 1, -1)) > > or the clearer but unaccountably sesquipedalian > > >>> wanted = [foo[i] for i in 3, 7, 1, -1] > >>> wanted = [foo[3], foo[7], foo[7], foo[-1]] > > Are these the most idiomatically pythonic forms? Or am I missing > something better? Firstly run "import this" at the python interactive interpreter to remind youself of the philosophical differences between perl and python. I think the philosophy of python is the major reason why it is such a good language. As I transitioned from perl to python it took me a while to let go of perlisms, and get used to writing a little bit more code (usually of the order of a few characters only) but which was much much clearer. Perl is full of cleverness which give you great pleasure to write as a programmer. However when you or someone else looks at that code later they don't get that same pleasure - horror is more likely the reaction! Python just isn't like that. I'd probably write wanted = foo[3], foo[7], foo[1], foo[-1] (assuming you didn't mind having a tuple rather than a list) or maybe this wanted = [ foo[i] for i in 3, 7, 1, -1 ] However I can't think of the last time I wanted to do this - array elements having individual purposes are usually a sign that you should be using a different data structure. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Lexical scope: converting Perl to Python
Andrew Savige wrote:
>
> I'd like to convert the following Perl code to Python:
>
> use strict;
> {
> my %private_hash = ( A=>42, B=>69 );
> sub public_fn {
> my $param = shift;
> return $private_hash{$param};
> }
> }
> print public_fn("A"); # good: prints 42
> my $x = $private_hash{"A"}; # error: good, hash not in scope
>
> The real code is more complex; the above is a simplified example.
>
> Notice that this code uses Perl's lexical scope to hide the
> %private_hash variable, but not the public_fn() function.
>
> While I could convert this code to the following Python code:
>
> private_hash = dict( A=42, B=69 )
> def public_fn(param):
> return private_hash[param]
> print public_fn("A") # good: prints 42
> x = private_hash["A"] # works: oops, hash is in scope
>
> I'm not happy with that because I'd like to limit the scope of the
> private_hash variable so that it is known only inside public_fn.
>
> Of course, I could hide the hash like so:
>
> def public_fn(param):
> private_hash = dict( A=42, B=69 )
> return private_hash[param]
>
> yet I'm not happy with that either because of the repeated
> initialization the hash each time the function is called.
>
> What is the Pythonic equivalent of Perl's lexical scope, as
> illustrated by the code snippet above?
Either
_private_hash = dict( A=42, B=69)
def public_fn(param):
return _private_hash[param]
Or
def public_fn(param, _private_hash = dict( A=42, B=69)):
return _private_hash[param]
Is probably the pythonic equivalents. Note that private_hash starts
with an underscore which means it won't be exported from a module by
default and it is a convention that it is private and shouldn't be
fiddled with. I'd probably go with the latter of the two examples.
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: unladen swallow: python and llvm
Stefan Behnel wrote: > Nick Craig-Wood wrote: > > Luis M González wrote: > >> I am very excited by this project (as well as by pypy) and I read all > >> their plan, which looks quite practical and impressive. > >> But I must confess that I can't understand why LLVM is so great for > >> python and why it will make a difference. > > > > CPython uses a C compiler to compile the python code (written in C) > > into native machine code. > > That would be Cython: compile Python code to (optimised) C code and then > run a C compiler over that to get native machine code. Actually I meant "compile the python source code" - sorry if I wasn't clear! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: setting program name, like $0= in perl?
m...@pixar.com wrote:
> I'm sure this is a FAQ, but I certainly haven't been able
> to find an answer.
>
> Is it possible to set the program name as seen by the
> operating system or lower-level libraries?
>
> I'm connecting to a database, and the runtime helpfully
> sends some information to the server, such as username,
> pid, and program name.
>
> Unfortunately, all my python programs get the name
> '/usr/bin/python', and I would like to force that to
> be the names of the individual scripts.
You can use this module
http://code.google.com/p/procname/
Just for fun I made a pure python version using ctypes
#!/usr/bin/python
"""
Attempt to set the process name with ctypes
"""
import os
from subprocess import Popen, PIPE
from ctypes import pythonapi, c_int, c_char_p, POINTER, addressof, pointer,
CDLL, memmove, memset
from ctypes.util import find_library
Py_GetArgcArgv = pythonapi.Py_GetArgcArgv
c_lib = CDLL(find_library("c"))
PR_SET_NAME = 15# linux specific
argc_t = POINTER(c_char_p)
Py_GetArgcArgv.restype = None
Py_GetArgcArgv.argtypes = [POINTER(c_int), POINTER(argc_t)]
def set_name(name):
argv = c_int(0)
argc = argc_t()
Py_GetArgcArgv(argv, pointer(argc))
name0 = name+"\0"
memset(argc.contents, 0, 256) # could do this better!
memmove(argc.contents, name0, len(name0))
# prctl doesn't seem to be needed on linux?
c_lib.prctl(PR_SET_NAME, name+"\0", 0, 0, 0)
def ps():
print Popen(["ps", "v", str(os.getpid())], stdout=PIPE).communicate()[0]
def main():
print "Before"
ps()
set_name("sausage")
print "After"
ps()
if __name__ == "__main__":
main()
This prints
$ ./procname.py
Before
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
9159 pts/7S+ 0:00 0 1000 5551 3404 0.1 /usr/bin/python
./procname.py
After
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
9159 pts/7S+ 0:00 0 1000 5551 3420 0.1 sausage
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: unladen swallow: python and llvm
Neuruss wrote: > ok, let me see if I got it: > The Python vm is written in c, and generates its own bitecodes which > in turn get translated to machine code (one at a time). > Unladen Swallow aims to replace this vm by one compiled with the llvm > compiler, which I guess will generate different bytecodes, and in > addition, supplies a jit for free. Is that correct? Pretty good I think! > It's confussing to think about a compiler which is also a virtual > machine, which also has a jit... Well the compiler is actually gcc with a llvm opcodes backend. > Another thing that I don't understand is about the "upfront" > compilation. > Actually, the project plan doesn't mention it, but I read a comment on > pypy's blog about a pycon presentation, where they said it would be > upfront compilation (?). What does it mean? I don't know, I'm afraid. > I guess it has nothing to do with the v8 strategy, because unladen > swallow will be a virtual machine, while v8 compiles everything to > machine code on the first run. But I still wonder what this mean and > how this is different. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: can it be shorter?
tsangpo wrote:
>
> "kj" 写入消息 news:h0e3p9$85...@reader1.panix.com...
> > In "tsangpo"
> > writes:
> >
> >>I want to ensure that the url ends with a '/', now I have to do thisa like
> >>below.
> >>url = url + '' if url[-1] == '/' else '/'
> >
> >>Is there a better way?
> >
> > It's a pity that in python regexes are an "extra", as it were.
> > Otherwise I'd propose:
> >
> > url = re.sub("/?$", "/", url)
> >
> > kynn (lowest-of-the-low python noob)
>
> look nice, but:
>
> >>> re.sub('/?$/', '/', 'aaabbb')
> 'aaabbb'
>
> has no effect. what a pity.
That is because you typoed what kynn wrote.
>>> re.sub('/?$', '/', 'aaabbb')
'aaabbb/'
>>>
That solution is very perl-ish I'd say, IMHO
if not url.endswith("/"):
url += "/"
is much more pythonic and immediately readable. In fact even someone
who doesn't know python could understand what it does, unlike the
regexp solution which requires a little bit of thought.
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: openhook
Gaudha wrote: > Can anybody tell me what is meant by 'openhook' ? http://docs.python.org/library/fileinput.html?highlight=openhook Maybe ;-) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Error in linalg.inv ??
Ajith Kumar wrote:
> I ran the following code (Using Debian 5.0)
>
> from numpy import *
> a = arange(1.,10.)
> b = reshape(a, [3,3])
> c = linalg.inv(b)
> print b
> print c
> print dot(b,c)
> print dot(c,b)
>
> And the result is
>
> [[ 1. 2. 3.]
> [ 4. 5. 6.]
> [ 7. 8. 9.]]
>
> [[ 3.15221191e+15 -6.30442381e+15 3.15221191e+15]
> [ -6.30442381e+15 1.26088476e+16 -6.30442381e+15]
> [ 3.15221191e+15 -6.30442381e+15 3.15221191e+15]]
>
> [[-0.5 -1. -1. ]
> [-1. -2. 2. ]
> [-1.5 -3. 1. ]]
>
> [[ 5.5 8. 10.5]
> [ 3.0. -3. ]
> [ -1.0. -3. ]]
>
> NOT the identity matrix. Any help ?
The matrix you are trying to invert is singular (can't be inverted),
ie its determinant is zero.
>> a = arange(1.,10.)
>>> b = reshape(a, [3,3])
>>> linalg.det(b)
-9.5171266700777579e-16
>>>
Which is zero but with a bit of rounding errors which I guess numpy
doesn't notice.
Double checking like this
>>> a,b,c,d,e,f,g,h,i=range(1,10)
>>> a*e*i - a*f*h - b*d*i + b*f*g + c*d*h - c*e*g
0
>>>
So I guess it is a bug that numpy didn't throw
numpy.linalg.linalg.LinAlgError("Singular matrix")
Like it does normally
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: unladen swallow: python and llvm
Luis M González wrote: > I am very excited by this project (as well as by pypy) and I read all > their plan, which looks quite practical and impressive. > But I must confess that I can't understand why LLVM is so great for > python and why it will make a difference. CPython uses a C compiler to compile the python code (written in C) into native machine code. unladen-swallow uses an llvm-specific C compiler to compile the CPython code (written in C) into LLVM opcodes. The LLVM virtual machine executes those LLVM opcodes. The LLVM virtual machine also has a JIT (just in time compiler) which converts the LLVM op-codes into native machine code. So both CPython and unladen-swallow compile C code into native machine code in different ways. So why use LLVM? This enables unladen swallow to modify the python virtual machine to target LLVM instead of the python vm opcodes. These can then be run using the LLVM JIT as native machine code and hence run all python code much faster. The unladen swallow team have a lot more ideas for optimisations, but this seems to be the main one. It is an interesting idea for a number of reasons, the main one as far as I'm concerned is that it is more of a port of CPython to a new architecture than a complete re-invention of python (like PyPy / IronPython / jython) so stands a chance of being merged back into CPython. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: What text editor is everyone using for Python
Ben Finney wrote: > Emile van Sebille writes: > > > On 6/4/2009 3:19 PM Lawrence D'Oliveiro said... > > > In message , Nick Craig- > > > Wood wrote: > > > > > >> You quit emacs with Ctrl-X Ctrl-C. > > > > > > That's "save-buffers-kill-emacs". If you don't want to save buffers, > > > the exit sequence is alt-tilde, f, e. > > This is an invocation of the menu system, driven by the keyboard. (Also, > it's not Alt+tilde (which would be Alt+Shift+`), it's Alt+` i.e. no > Shift.) It's an alternate command, and IMO is just adding confusion to > the discussion. Also, according to my emacs e==>Exit Emacs (C-x C-c) so Alt-` f e is exactly the same as Ctrl-x Ctrl-c anyway! If the OP really want to quit emacs without being prompted to save any buffes then run the 'kill-emacs' command which isn't bound to a key by default. You would do this with Alt-X kill-emacs But the fact that it isn't bound to a key by default means that it isn't recommended (and I've never used it in 10 years of using emacs!) - just use Ctrl-X Ctrl-C as Richard Stallman intended ;-) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: What text editor is everyone using for Python
Steven D'Aprano wrote: > On Tue, 02 Jun 2009 10:54:48 +1200, Lawrence D'Oliveiro wrote: > > > In message , Albert van der Horst wrote: > > > >> An indication of how one can see one is in emacs is also appreciated. > > > > How about, hit CTRL/G and see if the word "Quit" appears somewhere. > > Ah, one has to love user interfaces designed with mnemonic keyboard > commands so as to minimize the burden of rote learning on the user. > Presumably it is G for "Get me the frack outta here!". > > Having noted that the word "Quit" does appear, how do you then *actually* > Quit? Apart from taunting the user, what is it that Ctrl-G is actually > doing when it displays the word "Quit" in what seems to be some sort of > status bar? I love the idea of emacs taunting the user - it is like all that lisp code suddenly became self aware and decided to join in ;-) Ctrl-G is the emacs interrupt sequence. It cancels what you are doing. Kind of like Ctrl-C in the shell. You quit emacs with Ctrl-X Ctrl-C. Save a document with Ctrl-X Ctrl-S that that is probably enough for the emacs survival guide! If you run emacs in a windowing environment (eg X-Windows or Windows) you actually get menus you can choose save and quit off! Anyway, wrenching the thread back on topic - I use emacs for all my python editing needs (on linux, windows and mac) with pymacs or python-mode (depending on distro). I use the subversion integration extensively. The interactive features of emacs are really useful for testing python code - even more useful than the python interactive prompt for bits of code longer than a line or too. (Open a new window in python mode, type stuff, press Ctrl-C Ctrl-C and have the output shown in a different window. If you messed up, clicking on the error will put the cursor in the right place in the code). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Get the hard disk hardware serial number
MRAB wrote: > Jorge wrote: > > I need to know how to get the hardware serial number of a hard disk in > > python. > > > For Windows, see http://www.daniweb.com/forums/thread187326.html For linux I'd run this and parse the results. # smartctl -i /dev/sda smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Model Family: Seagate Momentus 7200.2 Device Model: ST9200420AS Serial Number:7QW138AK Firmware Version: 3.AAA User Capacity:200,049,647,616 bytes Device is:In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is:Thu Jun 4 09:30:23 2009 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled According to the man page smartctl also runs under windows/mac/solaris etc -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: easiest way to plot x,y graphically during run-time?
Esmail wrote:
> Scott David Daniels wrote:
> > Esmail wrote:
> >> ... Tk seems a bit more complex .. but I really don't know much about
> >> it and its interface with Python to make any sort of judgments as
> >> to which option would be better.
> >
> > This should look pretty easy:
>
> Thanks Scott for taking the time to share this code with me, it
> will give me something to study. I'm not sure if I'd say it looks
> easy (but then again I am not very familiar with Tk :-)
Here is a demo with pygame...
import pygame
from pygame.locals import *
from random import randrange
size = width, height = 640, 480
background_colour = 0x00, 0x00, 0x00
foreground_colour = 0x51, 0xd0, 0x3c
def main():
pygame.init()
screen = pygame.display.set_mode(size, 0, 32)
pygame.display.set_caption("A test of Pygame - press Up and Down")
pygame.mouse.set_visible(0)
clock = pygame.time.Clock()
dots = [ (randrange(width), randrange(height)) for _ in range(100) ]
radius = 10
while True:
clock.tick(60)
for event in pygame.event.get():
if event.type == QUIT:
raise SystemExit(0)
elif event.type == KEYDOWN:
if event.key == K_ESCAPE:
raise SystemExit(0)
elif event.key == K_UP:
radius += 1
elif event.key == K_DOWN:
radius -= 1
screen.fill(background_colour)
for dot in dots:
pygame.draw.circle(screen, foreground_colour, dot, radius, 1)
dots = [ (dot[0]+randrange(-1,2), dot[1]+randrange(-1,2)) for dot in
dots ]
pygame.display.flip()
if __name__ == "__main__":
main()
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Using C++ and ctypes together: a vast conspiracy? ;)
Diez B. Roggisch wrote:
> Joseph Garvin schrieb:
> > So I was curious whether it's possible to use the ctypes module with
> > C++ and if so how difficult it is. I figure in principal it's possible
> > if ctypes knows about each compiler's name mangling scheme. So I
> > searched for "ctypes c++" on Google.
[snip]
> > More seriously -- how difficult is it to use ctypes instead of saying,
> > boost::python, and why isn't this in a FAQ somewhere? ;)
>
> Because it's much more needed than name-mangling. Name mangling is
> (amongst other things) one thing to prevent
> C++-inter-compiler-interoperability which results from differing C++ ABIs.
>
> I'm not an expert on this, but AFAIK no common ABI exists, especially
> not amongst VC++ & G++. Which means that to work for C++, ctypes would
> need to know about the internals of both compilers, potentially in
> several versions, and possibly without prior notice to changes.
Probably depends on how far you want to dig into C++. I'm sure it
will work fine for simple function calls, passing classes as anonymous
pointers etc. If you want to dig into virtual classes with multiple
bases or the STL then you are probably into the territory you
describe.
That said I've used C++ with ctypes loads of times, but I always wrap
the exported stuff in extern "C" { } blocks.
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Ah, ctypes
David Bolen wrote: > Nick Craig-Wood writes: > > > ctypes could potentially note that function types don't have enough > > references to them when passed in as arguments to C functions? It > > might slow it down microscopically but it would fix this problem. > > Except that ctypes can't know the lifetime needed for the callbacks. If > the callbacks are only used while the called function is executing (say, > perhaps for a progress indicator or internal completion callback) then > it's safe to create the function wrapper just within the function > call. Good point... However I wouldn't mind if ctypes emitted a warning or even threw an exception in this case too though as the other case is so invidious. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Ah, ctypes
Lawrence D'Oliveiro wrote: > I wrote some code months ago to output 1-bit-per-pixel PNG files from > Python, doing direct calls to libpng using ctypes, because PIL didn't give > me sufficient control over colour tables and pixel depths. > > I thought the code was working fine. I left it aside for some months, came > back to it a week or two ago, and found it was crashing. I was creating > CFUNCTYPE objects to do callbacks to my own I/O routines [snip] > Make sure you keep references to CFUNCTYPE objects as long as they are > used from C code. ctypes doesn’t, and if you don’t, they may be garbage > collected, crashing your program when a callback is made. > > Yup, that was it. I changed my installation of the callbacks from [snip] As a ctypes user I found this an interesting story - thanks for posting it! ctypes could potentially note that function types don't have enough references to them when passed in as arguments to C functions? It might slow it down microscopically but it would fix this problem. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to Spawn a process with P_NOWAIT and pass it some data ?
Barak, Ron wrote:
> This is my first try at IPC in Python, and I would like to ask your help wi=
> th the following problem:
>
> I would like to spawn a process with P_NOWAIT, and pass some data to the ch=
> ild process.
>
> I created two scripts to try IPC (in a blocking way):
>
> $ cat subprocess_sender.py
> #!/usr/bin/env python
>
> import subprocess
>
> proc = subprocess.Popen(["python", "-u", "subprocess_receiver.py"],
> stdin=subprocess.PIPE, shell=True)
You don't need shell=True here I wouldn't have thought. It is a bad
idea in general.
> proc.communicate(input="this is sent from subprocess_sender.py")[0]
> proc.stdin.close()
> and
>
> $ cat subprocess_receiver.py
> #!/usr/bin/env python
>
> import sys
>
> print sys.stdin.readline()
>
> These scripts intercommunicate nicely:
>
> $ python -u subprocess_sender.py
> this is sent from subprocess_sender.py
[snip]
> Can anyone suggest what is the correct way to implement P_NOWAIT and still
> be able to communicate with the child process ?
You've written it already! subprocess doesn't wait for a processes
until you call the communicate or wait methods.
If you want to communicate with the subprocess but not block waiting
for all of its output then use the file handle proc.stdout, eg
import subprocess
proc = subprocess.Popen(["python", "-u", "subprocess_receiver.py"],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0)
proc.stdin.write("10\n")
proc.stdin.write("this is sent from subprocess_sender.py\n")
proc.stdin.close()
while True:
line = proc.stdout.readline()
if not line:
break
print "Received %r" % line
proc.wait()
# subprocess_receiver.py
import sys
import time
times = int(sys.stdin.readline())
line = sys.stdin.readline().rstrip()
for x in range(times):
print "%d: %s" % (x, line)
time.sleep(1)
$ python subprocess_test3.py
Received '0: this is sent from subprocess_sender.py\n'
Received '1: this is sent from subprocess_sender.py\n'
Received '2: this is sent from subprocess_sender.py\n'
Received '3: this is sent from subprocess_sender.py\n'
Received '4: this is sent from subprocess_sender.py\n'
Received '5: this is sent from subprocess_sender.py\n'
Received '6: this is sent from subprocess_sender.py\n'
Received '7: this is sent from subprocess_sender.py\n'
Received '8: this is sent from subprocess_sender.py\n'
Received '9: this is sent from subprocess_sender.py\n'
Received '10: this is sent from subprocess_sender.py\n'
(printed with a 1 second pause between each line)
----
If you want to interact with a subprocess (eg send, receive, send,
receive) then use the pexpect module - buffering in subprocess will
cause you nothing but pain otherwise!
> (Or, is there a way to create a subprocess.Popen object from what I assume =
> is the process handle integer ?)
Errr, not as far as I know.
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Adding a Par construct to Python?
Steven D'Aprano wrote:
> On Tue, 19 May 2009 05:52:04 -0500, Grant Edwards wrote:
>
> > On 2009-05-19, Steven D'Aprano
> > wrote:
> >> On Mon, 18 May 2009 02:27:06 -0700, jeremy wrote:
> >>
> >>> Let me clarify what I think par, pmap, pfilter and preduce would mean
> >>> and how they would be implemented.
> >> [...]
> >>
> >> Just for fun, I've implemented a parallel-map function, and done a
> >> couple of tests. Comments, criticism and improvements welcome!
> >
> > My only comment would be that your "slow function" might not be a very
> > simulation for the general-case, since it uses time.sleep() which
> > releases the GIL:
>
>
> I didn't expect my code to magically overcome fundamental limitations of
> the CPython interpreter :)
>
>
>
> >> def f(arg): # Simulate a slow function.
> >> time.sleep(0.5)
> >> return 3*arg-2
> >
> > Any Python function that isn't calling a library function written in C
> > that releases the GIL won't show any speedup will it?
>
> Not necessarily. Here's another function, that uses a loop instead of
> sleep.
>
> def g(arg, SIZE=8*10**6):
> # Default SIZE is chosen so that on my machine, the loop
> # takes approximately 0.5 second.
> for x in xrange(SIZE):
> pass
> return 3*arg-2
>
>
> >>> setup = 'from __main__ import pmap, g; data = range(50)'
> >>> min(Timer('map(g, data)', setup).repeat(repeat=5, number=3))
> 65.093590974807739
> >>> min(Timer('pmap(g, data)', setup).repeat(repeat=5, number=3))
> 20.268381118774414
I don't think that can be right - that shows python working without
the GIL contention. So unless you ran it under IronPython?
Here is what happens when I run it under CPython 2.5 on my dual core
laptop. I made SIZE=10**6 because I got bored of waiting ;-)
map
9.85280895233
pmap
28.4256689548
So the pmap took nearly 3 times as long. I expect this is because the
task was divided into 5 sections each competing madly for the GIL.
I ran the same script under the latest jython beta which was very
interesting! pmap showing a slight improvement, and faster than
cPython!
$ jython2.5rc2/jython pmap.py
map
6.242000103
pmap
5.8881144
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Options for creating a statistics screen on a Mac with Python?
Diez B. Roggisch wrote: > Phillip B Oldham schrieb: > > I've come into possession of a mac mini and a large LCD tv at the > > office. I'd like to set it up in the corner to pull statistics from > > our various servers (load, uptimes, etc) and display them in a > > graphical format, full-screen, with a reasonable refresh rate (say > > every 30 seconds). There will be quite a few statistics, and we'll be > > adding new servers over time, and since there won't be the option to > > scroll a full-screen web-page won't work. > > > > What then would be my options to create a full-screen display of > > statistics, and an attractive graphical way, using Python on OS X? > > matplotlib should serve you well. Or you use some web-based solution > with a javascript-based chart library. This might be even a bit > easier. I'd use pygame for a really clean full screen display. Probably a bit more work, but you'll get something really cool at the end of it! here is how to use matplotlib on a pygame surface http://www.pygame.org/wiki/MatplotlibPygame -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Need advice on distributing small module
kj wrote: > The module has only one non-standard dependency, described by the > following code: > > if sys.version_info[:2] >= (2, 6): > import json > else: > import simplejson as json I think try: import json except ImportError: import simplejson as json Is more pythonic... You aren't relying on what came with particular python versions which may not be true in jython/ironpython/etc. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: itertools question
Neal Becker wrote: > Is there any canned iterator adaptor that will > > transform: > in = [1,2,3] > > into: > out = [(1,2,3,4), (5,6,7,8),...] > > That is, each time next() is called, a tuple of the next N items is > returned. This is my best effort... not using itertools as my brain doesn't seem to work that way! class Grouper(object): def __init__(self, n, i): self.n = n self.i = iter(i) def __iter__(self): while True: out = tuple(self.i.next() for _ in xrange(self.n)) if not out: break yield out g = Grouper(5, xrange(20)) print list(g) g = Grouper(4, xrange(19)) print list(g) Which produces [(0, 1, 2, 3, 4), (5, 6, 7, 8, 9), (10, 11, 12, 13, 14), (15, 16, 17, 18, 19)] [(0, 1, 2, 3), (4, 5, 6, 7), (8, 9, 10, 11), (12, 13, 14, 15), (16, 17, 18)] -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: issue with twisted and reactor. Can't stop reactor
Gabriel wrote: > Jean-Paul Calderone escribió: > > None of the reactors in Twisted are restartable. You can run and > stop them > > once. After you've stopped a reactor, you cannot run it again. This is > > the > > cause of your problem. > > > > Jean-Paul > > I see. > Is it possible to do what I want using twisted? or > I should found another way? You have to go twisted... Your program should call reactor.run() once and everything you need your program to do should be started off from twisted callbacks (eg timer callbacks or deferred objects). To get your program to do something immediately after it is started, use reactor.callLater() before calling reactor.run(). You can't mix and match programming styles with twisted - it is all asynchronous callbacks or nothing in my experience! That takes a bit of getting your head round at first. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: OOP & Abstract Classes
Adam Gaskins wrote:
> I am a fairly seasoned PHP developer (don't shoot, I'm changing teams!:) who
> is admittedly behind the curve with OOP. Like most who learned PHP, I
> started doing web app backend stuff, but I have moved to full blown windows
> apps in the last 6 months using Winbinder. As you may know Winbinder is
> essentially abandoned, and programming windows apps with Winbinder or even
> GTK-PHP is less than ideal. I also deal heavily in serial/rs-232
> communications which is a pain in PHP. Long story short, I'm tired of doing
> things in such a hackish manner and want to write applications that are
> cross platform (I'd like to get our production dept on linux eventually) and
> truely object oriented.
>
> So I was beginning to learn OOP for PHP, and it seemed to me that abstract
> classes were just right for my application. In my application I must
> communicate with several peices of test equipment that communicate via
> RS-232. Most use SCPI instructions, some do not and require low level
> communication.
>
> The way I understand abstract classes is that I could have a class that has
> all my abstract methods such as 'init', 'getMeasurement', 'setPressure',
> etc... then I could use this common interface to to control my different
> pieces of hardware (after I write a library for each of them).
>
> Is this a valid application for abstract classes? Or am I making life more
> complicated than it need be?
This sounds like a fine use for Abstract classes.
Python doesn't have language support for them directly, but they are
usually written like this
class DeviceBase(object):
def __init__(self):
raise NotImplementedError()
def getMeasusrement(self):
raise NotImplementedError()
def setPressure(self, pressure):
raise NotImplementedError()
Then subclass it for the real functionality
class RealDevice(DeviceBase):
def __init__(self):
self.pressure = 0
def getMeasusrement(self):
return 0
def setPressure(self, pressure):
self.pressure = pressure
However, I normally found that there is some functionality that is
common to all subclasses so there may be some real methods in your
DeviceBase class, eg
class DeviceBase(object):
def __init__(self, device):
self.device = device
def getMeasurement(self):
raise NotImplementedError()
def setPressure(self, pressure):
raise NotImplementedError()
def __repr__(self):
return "%s(%r)" % (self.__class__.__name__, self.device)
class RealDevice(DeviceBase):
def __init__(self, device):
DeviceBase.__init__(self, device)
self.pressure = 0
def getMeasurement(self):
return self.pressure
def setPressure(self, pressure):
self.pressure = pressure
Which looks like this when you test it
>>> base = DeviceBase("/dev/ttyS0")
>>> base
DeviceBase('/dev/ttyS0')
>>> base.getMeasurement()
Traceback (most recent call last):
File "", line 1, in
File "", line 5, in getMeasurement
NotImplementedError
>>> base.setPressure(14)
Traceback (most recent call last):
File "", line 1, in
File "", line 7, in setPressure
NotImplementedError
>>>
>>> real = RealDevice("/dev/ttyS1")
>>> real
RealDevice('/dev/ttyS1')
>>> real.getMeasurement()
0
>>> real.setPressure(14)
>>> real.getMeasurement()
14
>>>
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Wrapping comments
Rhodri James wrote: > On Sun, 10 May 2009 08:32:23 +0100, Tobias Weber wrote: > > > In article , > > Arnaud Delobelle wrote: > > > >> A simple Alt-Q will reformat everything nicely. > > > > Now that's something. Thanks! > > > > (still not gonna use software that doesn't let me type # because it's > > alt+3 on a UK layout; having to re-learn or configure that is just sick) > > What on earth are you talking about? '#' has its own key on a UK layout > (shared with '~', but you know what I mean), just to the left of the > RETURN key. Emacs is my editor of choice, and I've never once come > across anything like this. You probably haven't used MAC OS X then! I vnc to a mac and use emacs and I just can't type a #. "Ctrl-Q 43 Return" is my best effort! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: how GNU stow is complementary rather than alternative to distutils
Zooko Wilcox-O'Hearn wrote: > On May 10, 2009, at 11:18 AM, Martin v. Löwis wrote: > > > If GNU stow solves all your problems, why do you want to use > > easy_install in the first place? > > That's a good question. The answer is that there are two separate > jobs: building executables and putting them in a directory structure > of the appropriate shape for your system is one job, and installing > or uninstalling that tree into your system is another. GNU stow does > only the latter. > > The input to GNU stow is a set of executables, library files, etc., > in a directory tree that is of the right shape for your system. For > example, if you are on a Linux system, then your scripts all need to > be in $prefix/bin/, your shared libs should be in $prefix/lib, your > Python packages ought to be in $prefix/lib/python$x.$y/site- > packages/, etc. GNU stow is blissfully ignorant about all issues of > building binaries, and choosing where to place files, etc. -- that's > the job of the build system of the package, e.g. the "./configure -- > prefix=foo && make && make install" for most C packages, or the > "python ./setup.py install --prefix=foo" for Python packages using > distutils (footnote 1). > > Once GNU stow has the well-shaped directory which is the output of > the build process, then it follows a very dumb, completely reversible > (uninstallable) process of symlinking those files into the system > directory structure. Once you've got that well formed directory structure it is very easy to make it into a package (eg deb or rpm) so that idea is useful in general for package managers, not just stow. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Q's on my first python script
kj wrote:
>
> Below is my very firs python script.
>
> This was just a learning exercise; the script doesn't do anything
> terribly exciting: for an argument of the form YYMMDD (year, month,
> day) it prints out the corresponding string YYMMDDW, where W is a
> one-letter abbreviation for the day of the week. E.g.
>
> % wd 090511
> 090511M
>
> The script runs OK, but I can still see a few areas of improvement,
> for which I could use your advice.
>
> 1. The name of the BadArgument exception class defined in the script
> does not seem to me sufficiently specific. If one were to import
> the script in order to reuse its wkday_abbrev function, I'd like
> this exception's name to be more unequivocally tied to this
> script. What I'm looking for is something like a "namespace"
> for this script. What's the pythonic way to construct a namespace?
>
> 2. In some python modules I've seen the idiom
>
> if __name__ == "__main__":
># run some tests here
>
> I'd like to set up tests for this script, mostly to ensure that
> it handles the error cases properly, but I'm alread using the
> idiom above to actually run the script under normal operation.
> What's the typical python idiom for running tests on a *script*
> (as opposed to a module that is normally not supposed to be run
> directly)?
>
> 3. Still on the subject of testing, how does one capture in a
> variable the output that would normally have gone to stdout or
> stderr?
>
> 4. What's the python way to emit warnings? (The script below should
> warn the user that arguments after the first one are ignored.)
>
> 5. The variable wd is meant to be "global" to the script. In other
> languages I've programmed in I've seen some typographic convention
> used for the name of such variables (e.g. all caps) to signal
> this widened scope. Does python have such a convention?
>
> Any comments/suggestions on these questions, or anything else about
> the script, would be much appreciated.
To be honest, my first try at this script probably would have looked
quite like yours! Once it was working I'd have tidied it as below.
* Put into a class to stop spread of globals
* WD made class member and upper case
* 4 space indents as per PEP8
* remove lamda as it isn't needed
* only try:/except: the minimum possible amount of code
* new name for exception
from optparse import OptionParser
import re
import datetime
import sys
class BadDateString(Exception):
"""Malformed date string"""
class Main(object):
"""Script to implement weekday"""
WD = ("M", "T", "W", "H", "F", "S", "U")
def wkday_abbrev(self, date_string):
mm = re.match("(\d{2})(\d{2})(\d{2})\Z", date_string)
if not mm:
raise BadDateString("Couldn't match date string from %r" %
date_string)
y, m, d = map(int, mm.groups())
if y < 38:
y = y + 1900
else:
y = y + 2000
try:
return self.WD[datetime.datetime(y, m, d).weekday()]
except ValueError:
raise BadDateString("Bad date in %r" % date_string)
def __init__(self):
usage = '''Usage: %prog [options] YYMMDD
%prog -h|--help
'''
parser = OptionParser(usage=usage)
parser.add_option("-n", "--no-newline", dest="nonl",
action="store_true", help="omit newline in output")
(options, args) = parser.parse_args();
try:
weekday = self.wkday_abbrev(args[0])
except BadDateString, e:
print usage
print e
sys.exit(1)
sys.stdout.write("%s%s" % (args[0], weekday))
if not options.nonl:
print
if __name__ == "__main__":
Main()
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Learning C++ for Python Development
joshua.pea...@gmail.com wrote: > I am a recovering C# web developer who has recently picked up Django > and I'm loving it. > > I would eventually like to get a job as a Django/Python developer. It > seems that many Python jobs require that you also be a C++ developer. > While I want to remain primarily a web developer, I don't want be > stuck doing CRUD applications, so I would like to learn C++ for Python > development. I have taken two basic programming courses in straight up > C++, no STL, Boost or anything like that, but I do have a very basic > knowledge of the language. > > Can you folks suggest some Python packages which use C++ and are > relevant for web development, so I can dig through the source and > contribute in some way? > > Or, just give me some general advice on learning C++ for Python? I'm guessing that people who advertise for C++ and Python are embedding python into C++ not extending python with C++. Having done both things, I can say that embedding python into C++ is not too much trouble, but that it is slightly easier to extend python with C rather than C++ since the python executable is a C program not a C++ program. I'd start by reading this http://docs.python.org/extending/ And try some of the examples. I'd also read all about ctypes http://docs.python.org/library/ctypes.html Which has saved me a huge amount of time in both the embedding and extending cases! One of the apps I've worked on is C++ but can have "drivers" written in python run by an embedded python interpreter. These drivers can call back into the C++ code, and the C++ can call into the drivers. That is done with a amount of C++ to load the drivers and to export the C++ symbols into the python code at runtime with ctypes. A bit of C++ implements the shims for the callbacks from python -> C++ (which are exported by ctypes). > P.S. I want to develop on Linux not Windows. Should be just the same on both. Once you've made your setup.py (for extending python) the build process will work on all supported architectures. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: unicode bit me
anuraguni...@yahoo.com wrote:
> First of all thanks everybody for putting time with my confusing post
> and I apologize for not being clear after so many efforts.
>
> here is my last try (you are free to ignore my request for free
> advice)
>
> # -*- coding: utf-8 -*-
>
> class A(object):
>
> def __unicode__(self):
> return u"©au"
>
> def __repr__(self):
> return unicode(self).encode("utf-8")
>
> __str__ = __repr__
>
> a = A()
> u1 = unicode(a)
> u2 = unicode([a])
>
> now I am not using print so that doesn't matter stdout can print
> unicode or not
> my naive question is line u2 = unicode([a]) throws
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
> 1: ordinal not in range(128)
>
> shouldn't list class call unicode on its elements?
You mean when you call unicode(a_list) it should unicode() on each of
the elements to build the resultq?
Yes that does seem sensible, however list doesn't have a __unicode__
method at all so I guess it is falling back to using __str__ on each
element, and which explains your problem exactly.
If you try your example on python 3 then you don't need the
__unicode__ method at all (all strings are unicode) and you won't have
the problem I predict. (I haven't got a python 3 in front of me at the
moment to test.)
So I doubt you'll find the momentum to fix this since unicode and str
integration was the main focus of python 3, but you could report a
bug. If you attach a patch to fix it - so much the better!
Here is my demonstration of the problem with python 2.5.2
>> class A(object):
... def __unicode__(self):
... return u"\N{COPYRIGHT SIGN}au"
... def __repr__(self):
... return unicode(self).encode("utf-8")
... __str__ = __repr__
...
>>> a = A()
>>> str(a)
'\xc2\xa9au'
>>> repr(a)
'\xc2\xa9au'
>>> unicode(a)
u'\xa9au'
>>> L=[a]
>>> str(L)
'[\xc2\xa9au]'
>>> repr(L)
'[\xc2\xa9au]'
>>> unicode(L)
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
1: ordinal not in range(128)
>>> unicode('[\xc2\xa9au]')
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position
1: ordinal not in range(128)
>>> L.__unicode__
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'list' object has no attribute '__unicode__'
>>> unicode(str(L),"utf-8")
u'[\xa9au]'
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Problems with datetime.datetime.strptime
Tuomas Vesterinen wrote:
> I hoped that I could get rid of my special module _strptime2 when
> porting to Python 3.0. But testing is a disappointment.
[snip]
> C :
> strftime('%a %b %e %H:%M:%S %Y')='Sat May 9 11:26:12 2009'
> strptime('Sat May 9 11:26:12 2009','%a %b %e %H:%M:%S %Y')=
> 'e' is a bad directive in format '%a %b %e %H:%M:%S %Y'
[snip]
> What to do.
I'd start by checking out the code for python 3 trunk and seeing if I
could fix it. If I could I'd update the unit tests and the
documentation then submit the patch to the python bugtracker.
If I couldn't fix it then I'd report it as a bug.
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Python process utility (psutil) 0.1.2 released
Giampaolo Rodola' wrote: > psutil is a module providing an interface for retrieving information > on running processes and system utilization (CPU, memory) in a > portable way by using Python, implementing many functionalities > offered by tools like ps, top and Windows task manager. > It currently supports Linux, OS X, FreeBSD and Windows. Very nice! Maybe you should make Process() mean Process(os.getpid()) as I often want ask how much memory/cpu is my process using but rarely want to ask about other processes? It would be nice if "pydoc psutil" returned something useful also! You could do this by replacing your current __init__.py (which just contains "from _psutil import *") with _psutil.py -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How should I use grep from python?
Matthew Wilson wrote: > I'm writing a command-line application and I want to search through lots > of text files for a string. Instead of writing the python code to do > this, I want to use grep. > > This is the command I want to run: > > $ grep -l foo dir > > In other words, I want to list all files in the directory dir that > contain the string "foo". > > I'm looking for the "one obvious way to do it" and instead I found no > consensus. I could os.popen, commands.getstatusoutput, the subprocess > module, backticks, etc. backticks is some other language ;-) > As of May 2009, what is the recommended way to run an external process > like grep and capture STDOUT and the error code? This is the one true way now-a-days >>> from subprocess import Popen, PIPE >>> p = Popen(["ls", "-l"], stdout=PIPE) >>> for line in p.stdout: ... print line ... total 93332 -rw-r--r-- 1 ncw ncw 181 2007-10-18 14:01 - drwxr-xr-x 2 ncw ncw 4096 2007-08-29 22:56 10_files -rw-r--r-- 1 ncw ncw 124713 2007-08-29 22:56 10.html [snip] >>> p.wait() # returns the error code 0 >>> There was talk of removing the other methods from public use for 3.x. Not sure of the conclusion. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: object query assigned variable name?
warpcat wrote: > I've passed this around some other groups, and I'm being told > "probably not possible". But I thought I'd try here as well :) I > *did* search first, and found several similar threads, but they > quickly tangented into other specifics of the language that were a bit > over my head :) At any rate, here's a simple example, I'd love to > know if as shown, is somehow possible: > > Given an object: > > class Spam(object): > def __init__(self): > # stuff > > I'd like it to print, when instanced, something like this: > > >>> s = Spam() > I’m assigned to s! > > But it seems prohibitively hard (based on my web and forum searches) > for an object to know what variable name is has been assigned to when > created. Querying 'self' in __init__ returns a memory location, not > the variable name passed in. > > If you're wondering why I'm trying to figure this out, this is just > part of my continued learning of the language and pushing the bounds, > to see what is possible ;) > > Any thoughts? Read up on introspection and learn how to look up through the stack frames. When you've mastered that look for an object matching self in all the locals in those stack frames. That will give some kind of answer. I have no idea whether this will work - the keyboard of my phone is too small to produce a proof ;-) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: wxPython menu creation refactoring
alex wrote: > I am still trying to refactor a simple GUI basing on an example in > "wxPython In Action", "Listing 5.5 A refactored example" where the > menue creation is "automatized". I understand the problem (each > second for loop in "def createMenuData (self)" creates a distinct > menu) but I tried now for some evenings and do not get to any > possible solution. I am still learning Python and hope to use it > as a front end for my Fortran programs (subprocess and ctypes work > very well...) with the aim of increasing the Python part (text > parsing) in the future. The following code shows the problem and > works out of the DOS box. Any help would be greatly apreciated. [snip] > def createMenu(self, itemLabel, itemHandler): > menu = wx.Menu() > menuItem = menu.Append(wx.NewId(), itemLabel) > self.Bind(wx.EVT_MENU, itemHandler, menuItem) > return menu > > def createMenuData(self): > menuBar = wx.MenuBar() > for eachMenuData in self.menuData(): > menuLabel = eachMenuData[0] > for eachLabel, eachHandler in eachMenuData[1:]: > menuBar.Append(self.createMenu(eachLabel, > eachHandler), menuLabel) > self.SetMenuBar(menuBar) I think you want this instead def createMenuData(self): menuBar = wx.MenuBar() for eachMenuData in self.menuData(): menuLabel = eachMenuData[0] submenu = wx.Menu() for eachLabel, eachHandler in eachMenuData[1:]: item = wx.MenuItem(submenu, -1, eachLabel) submenu.AppendItem(item) self.Bind(wx.EVT_MENU, eachHandler, item) menuBar.Append(submenu, menuLabel) self.SetMenuBar(menuBar) That is the way I normally do it anyway! You create the submenu as a seperate menu then attach it to the menuBar with the label. Note there is a wxpython list also which you may get more help in! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: bug with os.rename in 2.4.1?
Steven D'Aprano wrote: > On Tue, 28 Apr 2009 14:30:02 -0500, Nick Craig-Wood wrote: > > > t123 wrote: > >> It's running on solaris 9. Here is some of the code. It's actually > >> at the beginning of the job. The files are ftp'd over. The first > >> thing that happens is that the files get renamed before any processing > >> of the file. And when it fails, it always fails at the first file, > >> comm.dat. What I can't understand is why the inconsistent behavior. > >> > >> try: > >> if os.path.exists(paths.xferin_dir+'/COMM.DAT'): > >>os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/ > > > > This code is inherently racy... What if two copies of your code started > > simultaneously? They might both run the os.path.exists but only one > > will succeed in the os.rename. > > > > You could write instead > > > > try: > > os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/ > COMM.DAT'+'.0') > > except OSError: > > pass > > > > Which isn't racy. > > The race condition is still there. The only difference is that in the > first case it fails noisily, with an exception, and in the second it > fails quietly and does nothing. I'd argue that since os.rename implements the syscall rename() and that is defined to be atomic (give or take) then the above is atomic and can't possibly be racy. Wikipedia's definition :- A race condition or race hazard is a flaw in a system or process whereby the output and/or result of the process is unexpectedly and critically dependent on the sequence or timing of other events. The term originates with the idea of two signals racing each other to influence the output first. The original example was clearly racy. I guess using the wikipedia definition, the fact that I've caught the exception makes it no longer unexpected and hence no longer racy. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: bug with os.rename in 2.4.1?
t123 wrote:
> It's running on solaris 9. Here is some of the code. It's actually
> at the beginning of the job. The files are ftp'd over. The first
> thing that happens is that the files get renamed before any processing
> of the file. And when it fails, it always fails at the first file,
> comm.dat. What I can't understand is why the inconsistent behavior.
>
> try:
> if os.path.exists(paths.xferin_dir+'/COMM.DAT'):
>os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/
This code is inherently racy... What if two copies of your code
started simultaneously? They might both run the os.path.exists but
only one will succeed in the os.rename.
You could write instead
try:
os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/COMM.DAT'+'.0')
except OSError:
pass
Which isn't racy. Or if you wanted to be more thorough
import errno
try:
os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/COMM.DAT'+'.0')
except OSError, e:
if e.errno != errno.ENOENT:
raise
The traceback should show the exact problem though.
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Large data arrays?
Ole Streicher wrote: > Hi Nick, > > Nick Craig-Wood writes: > > mmaps come out of your applications memory space, so out of that 3 GB > > limit. You don't need that much RAM of course but it does use up > > address space. > > Hmm. So I have no chance to use >= 2 of these arrays simultaniously? > > > Sorry don't know very much about numpy, but it occurs to me that you > > could have two copies of your mmapped array, one the transpose of the > > other which would then speed up the two access patterns enormously. > > That would be a solution, but it takes twice the amount of address > space (which seems already to be the limiting factor). In my case (1.6 > GB per array), I could even not use one array. You don't need them mapped at the same time so you could get away with just one copy mapped. Also you can map the array in parts and use dramatically less address space. > Also, I would need to fill two large files at program start: one for > each orientation (row-wise or column-wise). Depending on the input > data (which are also either row-wise or column-wise), the filling of > the array with opposite direction would take a lot of time because of > the inefficiencies. > > For that, using both directions probably would be not a good > solution. What I found is the "Morton layout" which uses a kind of > fractal interleaving and sound not that complicated. It sounds cool! > But I have no idea on how to turn it into a "numpy" style: can I > just extend from numpy.ndarray (or numpy.memmap), and which > functions/methods then need to be overwritten? The best would be > ofcourse that someone already did this before that I could use > without trapping in all these pitfalls which occur when one > implements a very generic algorithm. I'd start by writing a function which took (x, y) in array co-ordinates and transformed that into (z) remapped in the Morton layout. Then instead of accessing array[x][y] you access morton_array[f(x,y)]. That doesn't require any subclassing and is relatively easy to implement. I'd try that and see if it works first! Alternatively you could install a 64bit OS on your machine and use my scheme! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory footpring of python objects
BlueBird wrote: > I have a program that manages several thousands instances of one > object. To reduce memory > consumption, I want of course that specific object to have the > smallest memory footpring possible. > > I have a few ideas that I want to experiment with, like using > __slots__, using a tuple or using a dict. My > question is: how do I know the memory footprint of a given python > object ? I could not find any > builtin functions for this in the documentation. I managed to 1/3 the memory requirement of our program by using __slots__ on the most common class (several hundred thousand instances!). When doing these optimisations I ran a repeatable script and measured the total memory usage using the OS tools (top in my case). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Large data arrays?
Ole Streicher wrote: > for my application, I need to use quite large data arrays > (100.000 x 4000 values) with floating point numbers where I need a fast > row-wise and column-wise access (main case: return a column with the sum > over a number of selected rows, and vice versa). > > I would use the numpy array for that, but they seem to be > memory-resistent. So, one of these arrays would use about 1.6 GB > memory which far too much. So I was thinking about a memory mapped > file for that. As far as I understand, there is one in numpy. > > For this, I have two questions: > > 1. Are the "numpy.memmap" array unlimited in size (resp. only limited > by the maximal file size)? And are they part of the system's memory > limit (~3GB for 32bit systems)? mmaps come out of your applications memory space, so out of that 3 GB limit. You don't need that much RAM of course but it does use up address space. > 2. Since I need row-wise as well as column-wise access, a simple usage > of a big array as memory mapped file will probably lead to a very poor > performance, since one of them would need to read values splattered > around the whole file. Are there any "plug and play" solutions for > that? If not: what would be the best way to solve this problem? > Probably, one needs to use someting like the "Morton layout" for the > data. Would one then build a subclass of memmap (or ndarray?) that > implements this specific layout? How would one do that? (Sorry, I am > still a beginner with respect to python). Sorry don't know very much about numpy, but it occurs to me that you could have two copies of your mmapped array, one the transpose of the other which would then speed up the two access patterns enormously. You needn't mmap the two arrays (files) at the same time either. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: pyflakes, pylint, pychecker - and other tools
Colin J. Williams wrote: > Esmail wrote: > > What is the consensus of the Python community regarding these > > code checkers? > > > > In particular, are the stylistic recommendations that > > pylint makes considered sensible/valid? > > pylint seems a bit heavy handled, a bit > too much PEP 8, which was intended as a > guide, rather than a prescription. I've been very happy with pychecker. I found pylint a bit too fussy (rather like the original C lint!) Note that if you run pychecker from emacs (M-x compile, then "pychecker module_name") you can then click in its output window to go to the correct line of code. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: A Special Thanks
norseman wrote: > I'm one of those that tries to get an outline of the project and then > puts in code as things become clear. Once the basics are working > reasonably I go back and organize the thing for maintainability. Then > finish flushing it out. It is the one stage I dread the most. > > Why not organize it up front? Because I don't always have the whole pie > at the outset. > > In changing to Python I had a bigger learning curve than I realized at > the start. When I finally got my "pieces" accomplishing what I wanted > it became time to start looking at its structure. > > I did the cut and paste into a followable form and ran the basic to > check for the usual errors, omissions and outright flaws. This was a > major reorganization. > > IT RAN FLAWLESSLY THE FIRST TRY! UNBELIEVABLE! (at least for me) Python has a well deserved reputation for being executable pseudo-code. It is the only language I've ever used where when you write the program it is quite likely to work the first time. Python also converted me to using unit tests. If you add unit tests into your methodology above then when you re-organize (or refactor to use the modern jargon) the code you can be 100% sure that you didn't break anything which is a wonderful feeling. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: How to create a virtual serial port?
Grant Edwards wrote:
> On 2009-04-12, JanC wrote:
> > Grant Edwards wrote:
> >
> >> On 2009-04-10, Stuart Davenport wrote:
> >>
> >>> I am trying to work out if its possible, to create a virtual serial
> >>> port with Python?
> >>
> >> On Linux: no.
> >
> > I wonder if there is no way to emulate ptys from userspace?
>
> Didn't I just answer that question?
>
> On Linux: no.
Actually you could do it with an LD_PRELOAD library
Intercept open("/dev/ttyS0",...). You'd need to intercept ioctl(),
read(), write(), close() etc too.
Messy but possible.
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: interacting with an updatedb generated data file within python
birdsong wrote: > Does anybody have any recommendations on how to interact with the data > file that updatedb generates? I'm running through a file list in > sqlite that I want to check against the file system. updatedb is > pretty optimized for building an index and storing it, but I see no > way to query the db file other than calling locate itself. This would > require me to fork and exec for every single file I want to verify - > I'd be better off doing the stat myself in that case, but I'd really > rather let updatedb build the index for me. Hmm.. There are several different implementations of locate and I'm not sure they all use the same database. On this ubuntu machine the database is only root readable also. $ ls -l /var/lib/mlocate/mlocate.db -rw-r- 1 root mlocate 7013090 2009-04-14 07:54 /var/lib/mlocate/mlocate.db > I searched high and low for any sort of library that is well suited > for reading these data files, but I've found nothing for any language > other than the source for locate and updatedb itself. You can use this to extract the database from the locate database from subprocess import Popen, PIPE from time import time start = time() all_files = set() p = Popen(["locate", "*"], stdout=PIPE) for line in p.stdout: path = line[:-1] all_files.add(path) print "Found", len(all_files), "files in", time()-start, "seconds" This builds a set of all the files on the filesystem and prints Found 314492 files in 1.152987957 seconds on my laptop, using about 19 MB total memory You could easily enough put that into an sqlite table instead of a set(). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Best way to extract from regex in if statement
Paul McGuire wrote:
> On Apr 3, 9:26 pm, Paul Rubin <http://phr...@nospam.invalid> wrote:
> > bwgoudey writes:
> > > elif re.match("^DATASET:\s*(.+) ", line):
> > > m=re.match("^DATASET:\s*(.+) ", line)
> > > print m.group(1))
> >
> > Sometimes I like to make a special class that saves the result:
> >
> > class Reg(object): # illustrative code, not tested
> > def match(self, pattern, line):
> > self.result = re.match(pattern, line)
> > return self.result
> >
> I took this a little further, *and* lightly tested it too.
>
> Since this idiom makes repeated references to the input line, I added
> that to the constructor of the matching class.
>
> By using __call__, I made the created object callable, taking the RE
> expression as its lone argument and returning a boolean indicating
> match success or failure. The result of the re.match call is saved in
> self.matchresult.
>
> By using __getattr__, the created object proxies for the results of
> the re.match call.
>
> I think the resulting code looks pretty close to the original C or
> Perl idiom of cascading "elif (c=re_expr_match("..."))" blocks.
>
> (I thought about cacheing previously seen REs, or adding support for
> compiled REs instead of just strings - after all, this idiom usually
> occurs in a loop while iterating of some large body of text. It turns
> out that the re module already caches previously compiled REs, so I
> left my cacheing out in favor of that already being done in the std
> lib.)
>
>
> import re
>
> class REmatcher(object):
> def __init__(self,sourceline):
> self.line = sourceline
> def __call__(self, regexp):
> self.matchresult = re.match(regexp, self.line)
> self.success = self.matchresult is not None
> return self.success
> def __getattr__(self, attr):
> return getattr(self.matchresult, attr)
That is quite similar to the one I use...
"""
Matcher class encapsulating a call to re.search for ease of use in conditionals.
"""
import re
class Matcher(object):
"""
Matcher class
m = Matcher()
if m.search(r'add (\d+) (\d+)', line):
do_add(m[0], m[1])
elif m.search(r'mult (\d+) (\d+)', line):
do_mult(m[0], m[1])
elif m.search(r'help (\w+)', line):
show_help(m[0])
"""
def search(self, r, s):
"""
Do a regular expression search and return if it matched.
"""
self.value = re.search(r, s)
return self.value
def __getitem__(self, n):
"""
Return n'th matched () item.
Note so the first matched item will be matcher[0]
"""
return self.value.group(n+1)
def groups(self):
"""
Return all the matched () items.
"""
return self.value.groups()
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: HTML Generation
J Kenneth King wrote: > Stefan Behnel writes: > > See here for another example that uses lxml.html: > > > > http://codespeak.net/lxml/lxmlhtml.html#creating-html-with-the-e-factory > > > > Stefan > > Ah, looks good. Have never used nor finished the example I had given -- > only meant as inspiration. I'm not surprised it has been done by someone > else. Something more like what the OP wanted is HTMLgen. I'm not sure it is maintained any more, but I used it quite a lot a few years ago. Debian still have the source and a package here http://packages.debian.org/sid/python-htmlgen But I think its original website is gone. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: double/float precision question
TP wrote:
> Hi everybody,
>
> Try the following python statements:
>
> >>> "%.40f" % 0.222
> '0.098864108374982606619596'
> >>> float( 0.222)
> 0.1
>
> It seems the first result is the same than the following C program:
>
> #include
>
> int main(void)
> {
> double a = 0.222;
>
> printf( "%.40f\n", a );
> return 0;
> }
> #
>
> My problem is the following:
> * the precision "40" (for example) is given by the user, not by the
> programmer.
> * I want to use the string conversion facility with specifier "e", that
> yields number is scientific format; so I cannot apply float() on the result
> of "%.40e" % 0.222, I would lost the scientific
> format.
>
> Is there any means to obtain the full C double in Python, or should I limit
> the precision given by the user (and used in "%.*e") to the one of a Python
> float?
Python floats are actually C doubles (as you proved yourself with your
little test program).
Eg
>>> 1.+2.**-52
1.0002
>>> 1.+2.**-53
1.0
Indicating that python floats have about 52 bits of precision, so are
definitely what C calls doubles.
When you do
>>> float( 0.222)
0.1
Python prints as many decimal places as are significant in the answer.
This is covered in the FAQ
http://www.python.org/doc/faq/general/#why-are-floating-point-calculations-so-inaccurate
If you want more precision use the built in decimal module or the
third party gmpy module.
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: A design problem I met again and again.
一首诗 wrote: > But I think the first step to resolve a problem is to describe it. In > that way, I might find the answer myself :-) That is a great saying! To answer your original question, split your code up into sections that can be tested independently. If you can test code in a isolated way then it belongs in a class / module of its own. If you have a class that is too big, then factor independent classes out of it until it is the right size. That is easier said than done and may require some creativity on your part. It will pay dividends though as the level of abstraction in your program will rise. I've noticed some programmers think in big classes and some think in small classes. Train yourself to do the other thing and your programming will improve greatly! -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Ordered Sets
Aahz wrote: > I find the trick of using a Python list to store the doubly-linked > list difficult to understand (as opposed to the usual mechanism of a > node class). I understand why it was done (time and space > efficiency), but I also still feel emotionally that it's somewhat > sick and perverted. I probably would feel more comfortable if the > doubly-linked list were abstracted out and commented. Heck, the > whole thing needs unit tests. Hmmm... Lets compare the two. Here is the length three list $ python Python 2.5.2 (r252:60911, Jan 4 2009, 17:40:26) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> L = [] >>> for i in xrange(100): ... L.append([1,2,3]) ... >>> import os >>> os.getpid() 28134 >>> (from top) 28134 ncw 20 0 58860 53m 1900 S0 2.6 0:02.62 python vs a Node class with __slots__ (for efficiency) $ python Python 2.5.2 (r252:60911, Jan 4 2009, 17:40:26) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class Node(object): ... __slots__ = ["prev", "next", "this"] ... def __init__(self, prev, next, this): ... self.prev = prev ... self.next = next ... self.this = this ... >>> Node(1,2,3) <__main__.Node object at 0xb7e897cc> >>> Node(1,2,3).prev 1 >>> L = [] >>> for i in xrange(100): ... L.append(Node(1,2,3)) ... >>> import os >>> os.getpid() 28203 >>> (from top) 28203 ncw 20 0 43364 38m 1900 S0 1.9 0:04.41 python So the Node class actually takes less memory 38 Mbytes vs 53 Mbytes for the list. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Programming Python 4th Edition?
Esmail wrote: > prueba...@latinmail.com wrote: > > It isn't a introduction to the Python language like "Learning Python", > > it doesn't work as reference like "Python in a Nutshell", it doesn't > > contain short idiomatic code like "Python Cookbook". What you are left > > with is different application domains and how to apply Python to them. > > The book is excellent if you want to do Network, GUI, Databases, etc. > > but poor if you want to learn about Python the core language. The > > title of the book should be changed from "Programming Python" to > > "Applied Python" > > I appreciate you taking the time to post. I agree with what you say. > I guess what appeals to me is the nearly encyclopedic nature of the > book .. and I am curious about scripting with python, so it seems to > have some good material on it (though I think there are newer modules > now available for this). > > It's good to hear what others think about this book, and others > too. I read Programming Python as an experienced programmer and like you I enjoyed the encyclopedic nature of it. So if it appeals to you I'd say go for it! The fact that it doesn't use the latest version of python isn't a problem - python doesn't change very quickly and emphasises backwards compatibility, even for the jump to 3.x. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: New Python Runtime / Unladen Swallow
Adonis wrote: > Came across this article on Ars on a new LLVM (Low Level Virtual > Machine) JIT compiler for Python being built by Google: > > > http://arstechnica.com/open-source/news/2009/03/google-launches-project-to-boost-python-performance-by-5x.ars Interesting article and site. The projct is called Unladen Swallow. (I'd write a link but I can't copy and paste them on my phone!) The idea seems to be to gradually transform python (2.6) into using LLVM as its VM instead of the python VM it uses at the moment. They've already managed to compile python into LLVM (there is a LLVM version of gcc) and speed it up a bit. The next step is to retarget the python byte code generator into producing LLVM bytecode instead. The advantage of targetting LLVM is that it is cross platform and already has JIT and full compilers for many platforms. They plan to fold their work back into CPython when done too. Sounds like a project to keep an eye on! > Now the question is will this make Vista run faster? Nothing could do that ;-) -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Interfacing python and C
MRAB wrote:
> steve William wrote:
> > Hi All,
> >
> > I'm using SWIG for the first time and I am facing some problems with
> > user defined header files. I'm trying to use my own header file in a C
> > program which would be interfaced with python.
> >
> > The header file is test.h:
> > /#include
> >
> > int fact(int n) {
> > if (n <= 1) return 1;
> > else return n*fact(n-1);
> > }/
> >
> > The C program is test1.c:
> > /#include
> > #include "test.h"
> >
> > int calc_fact(int a)
> > {
> > return (fact(a));
> > }/
> >
> > The interface file is test1.i:
> > /%module test1
> >
> > %{
> > #include "stdio.h"
> > #include "test.h"
> > %}
> >
> > int calc_fact(int a);/
> >
> > The commands that I used to generate the wrappers are:
> > /swig -python test1.i
> > gcc -c test1.c test1_wrap.c -I/usr/include/python2.5
> > -I/usr/lib/python2.5/config
> > g++ -shared test1_wrap.o -o _test1.so/
gcc not g++ here
link test1.o also
gcc -shared test1.o test1_wrap.o -o test1.so
> > When I try to import test1, I get an error saying:
> > />>> import test1
> > Traceback (most recent call last):
> > File "", line 1, in
> > File "test1.py", line 22, in
> > import _test1
> > ImportError: ./_test1.so: undefined symbol: calc_fact/
Try nm or objdump on the .so to see what symbols are in it.
> > I'm not sure why this is happening because when I try without user
> > defined header file, it works. Also the error says that it does not
> > recognize /calc_fact/ which is a function that I want to access from
> > python and is declared in the interface file.
> >
> > Is there any specific way in which user defined headers need to be
> > declared in the interface file? Should the user defined header be placed
> > in the /usr/include directory?
> >
> > Any help on this is highly appreciated.
My advice to you is to compile the C stuff into a .so and use ctypes
instead of swig. You then write the interface code in python not C
and you'll have a lot more fun!
cython is very useful in this area too provided you don't mind an
extra dependency. If you are developing C code from scratch to
use with python, then write it in cython instead!
> Should you be putting a function body in a header file?
No
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to port Python to an ARM running NucleusPlus OS
Thomas Ng wrote: > I'm planning to port the Python to run on an embedded device running > Nucleus+ OS on ARM7 processor. > > Can anyone help to advise how to get started? > > Where I am now: I've downloaded the Python 2.6 from SVN and able to > compile it using VC++. But no idea how to proceed from here. Since I > am not familiar with Linux, I would prefer to do the porting work on > Windows. Well... Nucleus isn't Linux, it is a RTOS. If it has enough POSIX interfaces you probably can port python but I wouldn't have thought it would be easy. Try to compile python in the cross compiling environment and see what happens! However if you are running Nucleus with Linux and want to run python in the Linux bit of it then I'd suggest to use the packages available for the Linux side of it. (Eg if it is running debian then apt-get install python). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: C extension using GSL
Gabriel Genellina wrote:
> En Fri, 27 Mar 2009 03:10:06 -0300, jesse escribió:
>
> > I give up. I cannot find my memory leak! I'm hoping that someone out
> > there has come across something similar. Let me lay out the basic
> > setup:
> > [...]
> > 4) C: A PyList object, L, is created (new reference!). This will hold
> > the solution vector for the ODE
> > [...]
> > 7) C: Return L to Python with return Py_BuildValue("N", L).
>
> I don't know the "N" format, but if Py_BuildValue returns a new reference,
> you're leaking a reference to the L list.
> Why don't you return L directly?
>
> You can use sys.getrefcount in Python to see the reference count for an
> object:
>
> py> from sys import getrefcount as rc
> py> x = object()
> py> rc(x)
> 2 # the name x, and a temporary reference as parameter
> py> rc([])
> 1 # only the temporary reference
> py> x = y = []
> py> rc(x)
> 3
> py> x = ()
> py> rc(x)
> 954 # the empty tuple is shared
That reminds me, you can use the gc module to show all your objects
that are in use, which can help with memory leaks.
eg something like
http://code.activestate.com/recipes/457665/
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Code anntotations (copyright, autor, etc) in your code
mattia wrote: > Hi all, which are the usual comments that you put at the beginning of > your code to explain e.g. the author, the usage, the license etc? > > I've found useful someting like: > > #- > # Name:About.py > # Purpose: > # > # Author: > # > # Created: 2009 > # Copyright: (c) 2009 > # Licence: GPL > > #- > > others put something like > > __author__ = "Name Surname" > __year__ = 2009 > > What do you use? __version__ is suggested in PEP 8 http://www.python.org/dev/peps/pep-0008/ __author__, __date__ and __credits__ are both understood by pydoc but I haven't actually seen that in a document only in the source code! I think the others are just conventions and are not actually used by anything, but I'd be interested to be proved wrong! I tend to use __author__ = "Nick Craig-Wood " __version__ = "$Revision: 5034 $" __date__ = "$Date: 2009-02-03 16:50:01 + (Tue, 03 Feb 2009) $" __copyright__ = "Copyright (c) 2008 Nick Craig-Wood" With __version__ and __date__ being version control tags (in this case svn) Pydoc produces this from the above ---- [snip] DATA __author__ = 'Nick Craig-Wood ' __copyright__ = 'Copyright (c) 2008 Nick Craig-Wood' __date__ = '$Date: 2009-02-03 16:50:01 + (Tue, 03 Feb 2009) $' __version__ = '$Revision: 5034 $' VERSION 5034 DATE $Date: 2009-02-03 16:50:01 + (Tue, 03 Feb 2009) $ AUTHOR Nick Craig-Wood pydoc pydoc shows an example of what __credits__ looks like -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: c.l.py dead, news at 11 (was Re: Mangle function name with decorator?)
Aahz wrote: > Well, yes, but that's simply the nature of online fora (I originally > wrote "nature of Usenet", but I think it's more general than that). From > my POV, if you're going to call it a "decline", you need to provide more > evidence than some people leaving and others arriving. I think that the > sheer volume and quality of posts to c.l.py is evidence that c.l.py is > not declining. c.l.py is my favourite usenet group and has been for some time. I've been doing usenet for 16 years now! I enjoy reading about problems and problems solved. I enjoy helping people where I can and especially learning new things when helping people - I think it has contributed enormously to my development as a python programmer. When I ask a question of c.l.py I find the answers to be informative and enlightening. Even after quite a few years of python programing I'm still learning new things from c.l.py As a long time usenet user I find it easy to ignore the occasional flame wars. Posters with the wrong sort of attitude are brought gently into line by the majority. If usenet groups had ratings I'd give c.l.py 5 stars. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: C extension using GSL
jesse wrote:
> I give up. I cannot find my memory leak! I'm hoping that someone out
> there has come across something similar. Let me lay out the basic
> setup:
>
> I'm performing multiple simulations on a model. Each iteration
> involves solving a system of differential equations. For this I use
> the GNU Scientific Library (GSL) -- specifically the rk4imp ODE
> solver. After the ODE is solved the array is returned to python and is
> analyzed. As you may have guessed, my problem is that over the course
> of the iterations the memory keeps climbing until python crashes.
>
> Note: *The extension does not keep running.* It returns object (a
> list) and is done until the next iteration. AFAIK, any memory
> allocated during execution of the extension should be released.
> Question: Since the extension was run from within python is memory
> allocated within an extension part of python's heap?
No, on the normal C heap
> Would this have
> an adverse or unpredictable affect on any memory allocated during the
> running of the extension?
If the library passes you data and ownership of a heap block, you'll
need to free it.
> One hypothesis I have, since most other
> possibilities have been eliminated, is that GSL's creation of it's own
> data structures (eg., gsl_vector) is messing up Python's control of
> the heap. Is this possible?
Only if GSL is buggy
> If so, how would one be able to fix this
> issue?
Valgrind
> It may help some nice folks out there who are good enough to look at
> this post if I layout the flow, broken up by where stuff happens
> (i.e., in the Python code or C code):
>
> 1) Python: Set up the simulation and basic data structures (numpy
> arrays with initial conditions, coupling matrices for the ODE's, dicts
> with parameters, etc).
> 2) Python: Pass these to the C extension
> 3) C: Python objects passed in are converted to C arrays, floats,
> etc.
> 4) C: A PyList object, L, is created (new reference!). This will hold
> the solution vector for the ODE
> 5) C: Initialize GSL ODE solver and stepper functions. Solve the ODE,
> at each step use PyList_Append(L, current_state) to append the current
> state to L.
> 6) C: After the ODE solver finishes, free GSL objects, free coupling
> matrices, free last state vector.
> 7) C: Return L to Python with return Py_BuildValue("N", L).
> 8) Python: Convert returned list L to array A, delete L, work with A.
> 8.1) Python: Step 8) includes plotting. (I have a small suspicion that
> matplotlib holds onto lots of data, but I've used clf() and close(fig)
> on all plots, so I think I'm safe here. )
> 8.2) Python: save analysis results from A, save A. (At this point
> there should be no more use of A. In fact, at point 8) in the next
> iteration A is replaced by a new array.)
> 9) Python: Change any parameters or initial conditions and goto 1).
At every point a memory allocation is made check who owns the memory
and that it is freed through all possible error paths.
Valgrind will help you find the memory leaks. It works well once
you've jumped the flaming hoops of fire that setting it up is!
Another thing you can try is run your process untill it leaks loads,
then make it dump core. Examine the core dump with a hex editor and
see what it is full of! This technique works suprisingly often.
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Bash-like brace expansion
Peter Waller wrote: > Okay, I got fed up with there not being any (obvious) good examples of > how to do bash-like brace expansion in Python, so I wrote it myself. > Here it is for all to enjoy! Interesting! Might I suggest some unit tests? You can then test all the corner cases (unmatched brackets, empty brackets, etc) and be sure it works exactly as specified. doctest is cool for this kind of stuff. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: udp package header
mete wrote: > I got a problem. İ want to send udp package and get this package (server and > clinet ). it's easy to python but i want to look the udp header how can i > do ? There is pretty much nothing in a UDP packet header except from_address, to_address, from_port and to_port. You should already know to_address and to_port and socket.recv will give you the from_address and from_port socket.recvfrom(bufsize[, flags]) Receive data from the socket. The return value is a pair (string, address) where string is a string representing the data received and address is the address of the socket sending the data. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Using python 3 for scripting?
Alan G Isaac wrote: > On 3/22/2009 12:41 PM Chris Rebert apparently wrote: > > 2.6.1, the latest non-3.x release is probably best. Most libraries > > haven't been ported to 3.x yet, so Python 3 has yet to become > > widespread. > > This seems slightly optimistic to me. Until a week ago, there was > not a NumPy release for 2.6. There is still not a SciPy release > for 2.6. Most dismaying, the SimpleParse guys need a little help > compiling SimpleParse for 2.6 and have not yet gotten that help. > (Is anyone listening?) > > So 2.5.4 is still perhaps the safest bet, even though it is more > awkward for writing code close to Python 3 syntax. I tend to target whatever is in Debian stable, which starting from this month is 2.5 (recently upgraded from 2.4). 2.6 or 3.x is nowhere to be seen in Debian stable, testing or unstable :-( -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone wrote:
> On Mon, 23 Mar 2009 05:30:04 -0500, Nick Craig-Wood
> wrote:
> >Jean-Paul Calderone wrote:
> > [snip]
> >>
> >> In the case of a TCP to serial forwarder, you don't actually have to
> >> implement either a producer or a consumer, since both the TCP connection
> >> and the serial connection are already both producers and consumers. All
> >> you need to do is hook them up to each other so that when the send buffer
> >> of one fills up, the other one gets paused, and when the buffer is empty
> >> again, it gets resumed.
> >
> >I eventually came up with this which seems to work, but I'm not sure
> >it is the best way of doing it as it had to mess about with the
> >twisted internals to get the number of bytes in the serial port
> >output buffer.
>
> This is sort of on the right track. Here's how to do it without
> touching implementation details:
Thank you for that!
See below for the complete prog with your suggested modifications.
That seems to solve the problem - I can see the writer pausing and
unpausing at the serial port rate.
write 16446
write 6430
pause producing
resume producing
write 65536
write 56724
pause producing
resume producing
write 65536
write 65536
pause producing
It has exposed a problem with the sender not throttling properly now,
but I guess that is progress!
Thanks for your help
Here is the complete program FYI with your suggested mods.
#!/usr/bin/python
"""Transfer data between a serial port and one (or more) TCP
connections.
options:
-h, --help:this help
-p, --port=PORT: port, a number, default = 0 or a device name
-b, --baud=BAUD: baudrate, default 115200
-t, --tcp=PORT: TCP port number, default 1234
-l, --log: log data streams to 'snifter-0', 'snifter-1'
-L, --log_name=NAME: log data streams to '-0', '-1'
"""
import sys
import getopt
from twisted.internet import reactor, protocol, serialport
from zope.interface import implements
from twisted.internet import protocol, interfaces
# FIXME set serial buffer size? SEND_LIMIT
class SerialPort(protocol.Protocol):
"""Create a serial port connection and pass data from it to a
known list of
TCP ports."""
def __init__(self, port, reactor, baudrate, log_name=None):
self.tcp_ports = []
self.serial = serialport.SerialPort(self, reactor, port,
baudrate, rtscts=0)
self.serial.registerProducer(self, True)
self.paused = False
self.log = None
if log_name is not None:
self.log = file('%s-0' % log_name, 'w')
def add_tcp(self, tcp_port):
"""Add a TCPPort to those receiving serial data."""
if self.paused:
tcp_port.transport.pauseProducing()
self.tcp_ports.append(tcp_port)
def del_tcp(self, tcp_port):
"""Remove a TCPPort from the those receiving serial data."""
self.tcp_ports.remove(tcp_port)
def write(self, data):
"""Write data to the serial port."""
self.serial.write(data)
if self.log:
self.log.write(data)
def pauseProducing(self):
"""Pause producing event"""
print "pause producing"
self.paused = True
for port in self.tcp_ports:
port.transport.pauseProducing()
def resumeProducing(self):
"""Resume producing event"""
print "resume producing"
self.paused = False
for port in self.tcp_ports:
port.transport.resumeProducing()
def stopProducing(self):
"""Stop producing event"""
print "stop producing"
def dataReceived(self, data):
"""Pass any received data to the list of TCPPorts."""
for tcp_port in self.tcp_ports:
tcp_port.write(data)
class TCPPort(protocol.Protocol):
"""Create a TCP server connection and pass data from it to the
serial port."""
def __init__(self, serial, log_name, index):
"""Add this TCPPort to the SerialPort."""
self.serial = serial
self.serial.add_tcp(self)
self.log = None
if log_name is not None:
self.log = file('%s-%d' % (log_name, index+1), 'w')
def __del__(self):
"""Remove this TCPPort from the SerialPort."""
self.serial.del_tcp(self)
def dataReceived(self, data):
"""Pass received data
Re: Async serial communication/threads sharing data
Jean-Paul Calderone wrote:
> On Sun, 22 Mar 2009 12:30:04 -0500, Nick Craig-Wood
> wrote:
> >I wrote a serial port to TCP proxy (with logging) with twisted. The
> >problem I had was that twisted serial ports didn't seem to have any
> >back pressure. By that I mean I could pump data into a 9600 baud
> >serial port at 10 Mbit/s. Twisted would then buffer the data for me
> >using 10s or 100s or Megabytes of RAM. No data would be lost, but
> >there would be hours of latency and my program would use up all my RAM
> >and explode.
> >
> >What I wanted to happen was for twisted to stop taking the data when
> >the serial port buffer was full and to only take the data at 9600
> >baud.
>
> This is what Twisted's producers and consumers let you do. There's a
> document covering these features:
>
> http://twistedmatrix.com/projects/core/documentation/howto/producers.html
>
> In the case of a TCP to serial forwarder, you don't actually have to
> implement either a producer or a consumer, since both the TCP connection
> and the serial connection are already both producers and consumers. All
> you need to do is hook them up to each other so that when the send buffer
> of one fills up, the other one gets paused, and when the buffer is empty
> again, it gets resumed.
I eventually came up with this which seems to work, but I'm not sure
it is the best way of doing it as it had to mess about with the
twisted internals to get the number of bytes in the serial port
output buffer.
class SerialPort(protocol.Protocol):
"""Create a serial port connection and pass data from it to a known list of
TCP ports."""
def __init__(self, port, reactor, baudrate, log_name=None):
self.tcp_ports = []
self.serial = serialport.SerialPort(self, reactor, port, baudrate,
rtscts=0)
self.log = None
if log_name is not None:
self.log = file('%s-0' % log_name, 'w')
def write(self, data):
"""Write data to the serial port."""
self.serial.write(data)
if self.log:
self.log.write(data)
self.throttle()
def throttle(self):
"""
Pause the inputs if there is too much data in the output buffer
"""
bytes_in_buffer = len(self.serial.dataBuffer) + self.serial._tempDataLen
too_full = bytes_in_buffer > 1024
for tcp_port in self.tcp_ports:
if too_full:
tcp_port.transport.pauseProducing()
else:
tcp_port.transport.resumeProducing()
if too_full:
reactor.callLater(0.1, self.throttle)
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
Jean-Paul Calderone wrote: > It's true that the serial port support in Twisted isn't the most used > feature. :) These days, serial ports are on the way out, I think. That > said, much of the way a serial port is used in Twisted is the same as the > way a TCP connection is used. This means that the Twisted documentation > for TCP connections is largely applicable to using serial ports. The only > major difference is how you set up the "connection". You can see examples > of using the serial port support here (one of them seems to suggest that > it won't work on Windows, but I think this is a mistake): > >http://twistedmatrix.com/projects/core/documentation/examples/gpsfix.py >http://twistedmatrix.com/projects/core/documentation/examples/mouse.py > > The 2nd to last line of each example shows how to "connect" to the serial > port. These basic documents describe how to implement a protocol. Though > in the context of TCP, the protocol implementation ideas apply to working > with serial ports as well: > >http://twistedmatrix.com/projects/core/documentation/howto/servers.html >http://twistedmatrix.com/projects/core/documentation/howto/clients.html > > You can ignore the parts about factories, since they're not used with serial > ports. I wrote a serial port to TCP proxy (with logging) with twisted. The problem I had was that twisted serial ports didn't seem to have any back pressure. By that I mean I could pump data into a 9600 baud serial port at 10 Mbit/s. Twisted would then buffer the data for me using 10s or 100s or Megabytes of RAM. No data would be lost, but there would be hours of latency and my program would use up all my RAM and explode. What I wanted to happen was for twisted to stop taking the data when the serial port buffer was full and to only take the data at 9600 baud. I never did solve that problem :-( -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: script files with python (instead of tcsh/bash)?
Esmail wrote:
> thanks for including the script, that really helps. Nice way of
> finding files.
Python has lots of useful stuff like that!
> Two quick questions:
>
> As a replacement for grep I would use the re module and its
> methods?
The re module works on strings not files, but basically yes.
Note that the re module uses a superset of the "grep -E" regexps, and
they are almost identical to those used in perl and php.
Here is a simple grep in python
#!/usr/bin/python
import re
import sys
import fileinput
pattern = sys.argv.pop(1)
for line in fileinput.input():
if re.search(pattern, line):
print line.rstrip()
Save in a file called grep.py then you can do
$ ./grep.py dizzy *.py
self.dizzy = 0
if self.dizzy:
$ ls | ./grep.py '\d{2}'
linux-image-2.6.24-21-eeepc_2.6.24-21.39eeepc1_i386.deb
linux-ubuntu-modules-2.6.24-21-eeepc_2.6.24-21.30eeepc6_i386.deb
> What about awk which I regularly use to extract fields based on position
> but not column number, what should I be using in Python to do the
> same?
I presume you mean something like this
... | awk '{print $2}'
In python, assuming you've got the line in the "line" variable, then
In python an equivalent of the above would be
import fileinput
for line in fileinput.input():
print line.split()[1]
Note that the fileinput module is really useful for making shell
command replacements!
> The other things I need to do consist of moving files, manipulating file
> names and piping outputs of one command to the next, so I'm digging into
> the documentation as much as I can.
Read up on the os module and the subprocess module. You'll find you
need to do much less piping with python as with shell because it has
almost everything you'll need built in.
Using built in functions is much quicker than fork()-ing an external
command too.
> So much to learn, so little time (but so much fun!)
;-)
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: 3.0 - bsddb removed
Sean wrote: > Anyone got any thoughts about what to use as a replacement. I need > something (like bsddb) which uses dictionary syntax to read and write an > underlying (fast!) btree or similar. sqlite. bsddb gave me no end of trouble with threads, but sqlite worked brilliantly. You would need to make a dictionary interface to sqlite, eg http://code.activestate.com/recipes/576638/ Or do something a bit simpler yourself. -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
