Re: IPv6 and Python
> Thanks a lot. I've been able to listen on ::1:21 after having
> installed IPv6 on both Windows and Linux thanks to your suggestions.
> I'd like to ask one more question: is it possible to bind my server on
> both IPv4 and IPv6 addresses (note: I do not use multiple threads or
> processes)?
You can, of course, open two sockets, and bind one for IPv4 and one for
IPv6. If you then want to use a single thread only, you need to use
poll/select, which will return when there is an incoming connection on
either socket. You then accept() the new connection for the respective
listen socket.
There is a notion of "mapped" IPv4 addresses, which allows to have a
single IPv6 server socket accept both IPv4 and IPv6 connections.
How this precisely works depends on the operating system.
On Linux, "mapped" addresses are the default. Open a v6 socket, and
connect to it through a v4 client, and it will just work. For this,
you need to specify the "any" server address, i.e.
py> s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
py> s.bind(('', 1234))
py> s.listen(100)
py> s.accept()
(, (':::77.132.228.129',
43491, 0, 0))
py> s.accept()
(, ('2001:6f8:900:a85::2',
40800, 0, 0))
The first connection was from a v4 client; the second connection from
a v6 client. To *disable* mapped addresses, you need to set the
IPV6_V6ONLY socket option to 1.
On Windows XP (including SP2), mapped addresses are not supported.
On Windows Vista, mapped addresses are supported, but you have
to explicitly *enable* a socket for mapped use, by setting
the IPV6_V6ONLY option to 0.
So in short, you should always set the IPV6_V6ONLY to 0, as the
system defaults vary. If that option is not defined in the socket
module, the system does not support mapped sockets.
Unfortunately, there is an exception to the last rule: on Vista,
the option is supported, but Python won't magically know what the
option numeric value is. So on Windows, you need to define
IPV6_V6ONLY yourself (the value is 27), and check whether the
setsockopt call succeeds.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: image matching algorithms
On Mar 14, 10:59 am, "Daniel Fetchinson" <[EMAIL PROTECTED]> wrote: > > > Since you seem to know quite a bit about this topic, what is your > > > opinion on the apparently 'generic' algorithm described here: > > >http://grail.cs.washington.edu/projects/query/? > > > So far it seems to me that it does what I'm asking for, it does even > > > more because it can take a hand drawn sample image and query the > > > database for similar photos. > > > > There is even a python implementation for it here: > > >http://members.tripod.com/~edcjones/pycode.html > > > > On the histogram method I agree that it won't work partly because of > > > what you say and partly because it is terribly slow since it's > > > comparing every single pixel. > > > I'm hardly the expert and can't answer authoritatively, but here's my 2c. > > > I can't comment as to the actual accuracy of the algorithm, since it > > will depend on your specific data set (set of photos). The algorithm is > > sensitive to spatial and luminance information (because of the YIQ > > colorspace), so there are simple ways in which it will fail. > > > The histogram method uses only color, but has a lot of numbers to > > compare. You may find the histogram method insensitive to spatial > > relations (a landscape with the mountain on the left and one with the > > mountain on the right) compared to the wavelet approach. > > > This is a relatively old paper, and I've seen other more recent image > > retrieval research using wavelets (some cases using only the > > high-frequency wavelets for "texture" information instead of the > > low-frequency ones used by this paper for "shape") and other information > > retrieval-related research using lossy compressed data as the features. > > If you have time, you may want to look at other research that cite this > > particular paper. > > > And just a thought: Instead of merely cutting off at m largest-wavelets, > > why not apply a quantization matrix to all the values? > > I'm not at all an expert, just started to look into image matching, so > I'm not quite sure what you mean. What's a quantization matrix in this > context? Hello, I am also looking for the solution to the same problem. Could you let me know if you have found something useful so far? I appreciate your response. Thanks a lot. Sengly -- http://mail.python.org/mailman/listinfo/python-list
Re: portable fork+exec/spawn
Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > >For jobs which require interactivity ie send input, receive output, > >send input, receive output, ... it doesn't work well. There isn't a > >good cross platform solution for this yet. pyexpect works well under > >unix and is hopefully being ported to windows soon. > > There is a good cross-platform solution, in fact. It's Twisted's > spawnProcess API, which works on POSIX and Windows, supports supports > sending and receiving, and doesn't have deadlock issues since it's > event-driven. It took me a while but I found the documentation on this eventually http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorProcess.html Looks interesting - I'll have to try it next time I'm reaching for pexpect Thanks Nick -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Help with pyserial and sending binary data?
Rich <[EMAIL PROTECTED]> wrote:
> I am working on a python library for sending and receiving data
> from a Subaru's ECU (the fuel injection computer) via the OBD-II
> port and an OBD to USB cable, with the Subaru Select Monitor
> protocol.
[snip]
> So I've been messing with it, and in as few lines as possible, this
> should in theory, send the init command to the ECU (0x80 0x10 0xF0
> 0x01 0xBF 0x40), and the ECU should return its ID to let you know
> that you can now chat (something like 0x80 0xF0 0x10 0x39 0xFF 0xA2
> 0x10 0x0F 0x1B 0x14 0x40 0x05 0x05 0x73 0xFA 0xEB ..):
>
> #--
> import serial, string
> output = " "
> ser = serial.Serial('/dev/ttyUSB0', 4800, 8, 'N', 1, timeout=1)
> ser.write(chr(0x80)+chr(0x10)+chr(0xF0)+chr(0x01)+chr(0xBF)+chr(0x40))
> while output != "":
> output = ser.read()
> print hex(ord(output))
> #--
The code looks OK.
With a constructor which takes as many arguments as Serial does I tend
to name them to avoid mistakes, eg
serial.Serial("/dev/ttyUSB0", baudrate=4800, bytesize=8, parity='N',
stopbits=1, timeout=1)
8N1 is documented as the default so you could then reduce it to the
following if you wanted
serial.Serial("/dev/ttyUSB0", baudrate=4800, timeout=1)
> The only problem is that when I send the ECU init command, it just
> echos back the data I sent it, which, from my understanding, means
> that I sent an invalid request packet.
My experience with these sort of protocols is that if you make a
packet error (wrong address, wrong checksum, wrong start byte) you'll
get no reply at all.
If you make a command error (use a command that isn't understood in a
valid packet) you'll get a properly formatted reply with an error
message in. This should start 0x80 0xF0 0x10 (from and to
reversed).
I'd suspect that if you are just receiving the data back then you have
got your serial cable wrong and have somehow looped tx and rx. Did
you try your cable with the other programs? Or possibly you are using
the wrong serial port - are you sure you haven't any other USB
serials? ls /dev/ttyUSB* on a udev system will show you. lsusb (as
root) is useful as is dmesg immediately after plugging the port in.
I do a lot of this sort of thing at work (not with cars though with
satellite equipment) and it is always the first packet and the first
response which is the hard part. After that it is usually plain
sailing!
--
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Newbie question - probably FAQ (but not exactly answered by regular FAQ)
Hi, I've gone through the list of "language differences" between 2.3 / 2.4 & 2.5 of CPython. I've spend around 2 weeks now, learning v2.5 of CPython, and I consider myself still very very newbie. So, unable to take a call as to how-important or desirable the newer language features are -- so whether to write my app for v2.5 of Python, versus, as few others on this list have recommended, i.e. to stick to v2.3 ?? Are the preformance improvements, and memory footprint / leak fix in 2.5 enough to justify moving to it ? What all do I stand to lose (or gain) by writing on v2.3 ?? I've a constraint due to which I might have to also write parts of my app (client side) in Jython (because I want to eventually ship Java -- yet benefit from the rapid development and clarity of Python). Would sticking to v2.3 in such a case be a better idea ? Suggestions with reasoning would be very helpful. -- regards, Banibrata http://www.linkedin.com/in/bdutta -- http://mail.python.org/mailman/listinfo/python-list
How to use a parameter in a class
I have the following class: # class Dummy(): value = 0 def __init__(self, thisValue): print thisValue self.value = thisValue value = thisValue def testing(self): print 'In test: %d' % self.value def returnValue(self): return self.value result = someFuntion(default = value) # But the last line does not work. I would like to do a call like: dummy = Dummy(thisValue = 12) And that someFunction gets a default value of 12. How can I do that? -- http://mail.python.org/mailman/listinfo/python-list
about bsddb module
How to deal with multiple databases in an file. I want to get the content of several databases. it's the code I wrote: [EMAIL PROTECTED] ~]$ python Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11) [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import bsddb >>> import os >>> import time >>> >>> db_file = 'native.db' >>> db = bsddb.db.DB() >>> db.open(db_file,bsddb.db.DB_UNKNOWN,bsddb.db.DB_RDONLY) >>> dbs = db.keys() >>> db.open(db_file,dbs[0],bsddb.db.DB_UNKNOWN,bsddb.db.DB_RDONLY) >>> db.keys() ['\x01\x00\x00\x00', '\x02\x00\x00\x00', '\x03\x00\x00\x00', '\x04\x00\x00\x00', '\x05\x00\x00\x00'] >>> db.open(db_file,dbs[1],bsddb.db.DB_UNKNOWN,bsddb.db.DB_RDONLY) the program stop here. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to use a parameter in a class
On May 3, 1:05 pm, Decebal <[EMAIL PROTECTED]> wrote: > I have the following class: > # > class Dummy(): > value = 0 > def __init__(self, thisValue): > print thisValue > self.value = thisValue > value = thisValue > > def testing(self): > print 'In test: %d' % self.value > > def returnValue(self): > return self.value > > result = someFuntion(default = value) > # > > But the last line does not work. > I would like to do a call like: > dummy = Dummy(thisValue = 12) > > And that someFunction gets a default value of 12. How can I do that? The line > result = someFuntion(default = value) is executed when the whole 'class' compound statement is executed, i.e. before the line that creates the 'dummy' instance. Moreover, this happens only once and not every time you create a new instance of the class. If you want 'result' to be a class attribute that is set every time you create a new instance move the line > result = someFuntion(default = value) in the __init__ constructor (you also have to assign some value to it before, just like the 'value' attribute and preffix it with Dummy. otherwise it'll be considered as local name of __init__): class Dummy(): value = 0 result = 0 def __init__(self, thisValue): print thisValue self.value = thisValue Dummy.value = thisValue Dummy.result = someFuntion(default = Dummy.value) def testing(self): ... -- http://mail.python.org/mailman/listinfo/python-list
Re: How to use a parameter in a class
On Sat, 03 May 2008 03:05:31 -0700, Decebal wrote: > I have the following class: > # > class Dummy(): > value = 0 > def __init__(self, thisValue): > print thisValue > self.value = thisValue > value = thisValue > > def testing(self): > print 'In test: %d' % self.value > > def returnValue(self): > return self.value > > result = someFuntion(default = value) > # > > But the last line does not work. > I would like to do a call like: > dummy = Dummy(thisValue = 12) > > And that someFunction gets a default value of 12. How can I do that? class Dummy(object): def __init__(self, thisValue): self.value = thisValue def someFunction(self, default=None): if default is None: default = self.value -- Ivan -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie question - probably FAQ (but not exactly answered by regular FAQ)
Banibrata Dutta <[EMAIL PROTECTED]> wrote: > I've gone through the list of "language differences" between 2.3 / 2.4 > & 2.5 of CPython. I've spend around 2 weeks now, learning v2.5 of > CPython, and I consider myself still very very newbie. So, unable to > take a call as to how-important or desirable the newer language > features are -- so whether to write my app for v2.5 of Python, versus, > as few others on this list have recommended, i.e. to stick to v2.3 ?? > Are the preformance improvements, and memory footprint / leak fix in > 2.5 enough to justify moving to it ? What all do I stand to lose (or > gain) by writing on v2.3 ?? If you are writing for 2.3 you are writing for 2.4 and 2.5 also. There are some nice things in 2.4 and 2.5 but nothing you really need IMHO. So I'd say writing in a 2.3 subset would be perfectly sensible. > I've a constraint due to which I might have to also write parts of my > app (client side) in Jython (because I want to eventually ship Java -- > yet benefit from the rapid development and clarity of Python). Would > sticking to v2.3 in such a case be a better idea ? Suggestions with > reasoning would be very helpful. Jython seems to be based off python 2.2 so you would be limited to 2.2 features in that case. No big deal in my opinion. -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Phyton module for Windows Event Viewer?
Hi all, Can someone point me in the right direction? I'm looking for a module to monitor the Windows Event Viewer. Thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: #!/usr/bin/env python vs. #!/usr/bin/python
In article <[EMAIL PROTECTED]>, Grant Edwards <[EMAIL PROTECTED]> wrote: > On 2008-05-02, D'Arcy J.M. Cain <[EMAIL PROTECTED]> wrote: > > On Sat, 03 May 2008 00:44:00 +1000 > > Ben Finney <[EMAIL PROTECTED]> wrote: > >> "D'Arcy J.M. Cain" <[EMAIL PROTECTED]> writes: > >> > As someone else pointed out, not all the world is Linux. > >> > >> It's a good thing I've never implied such to be the case. > > > > You haven't *said* it but you have definitely *implied* it. > > Installing Python in /usr/bin is not common. > > It is common. That's where it's installed by almost all Linux > distributions. MacOS X system python (or links to them) is in the same place. -- -- Lou Pecora -- http://mail.python.org/mailman/listinfo/python-list
Re: pygame.key.get_pressed[K_a], K_a is not defined!?
globalrev schrieb: On 2 Maj, 18:13, "Diez B. Roggisch" <[EMAIL PROTECTED]> wrote: globalrev schrieb: if pygame.key.get_pressed[K_a]: print "Muppet" K_a is not defined. but yes it is. why do i get this error? No it isn't - otherwise you wouldn't get this error, wouldn't you? What IS defined is pygame.K_a Or if you do from pygame import K_a then K_a is defined as well. Diez ok thanks if (key.get_pressed[K_t] and key.get_pressed[K_f]): print "Yup!" said int he tut it confused me. anyway i changed it and the list or dict from keygetpressed is all zeros but the K_a always is true whether i push it or not. most probably because key.get_pressed[K_t] is bogus. It's a method you need to call that returns a list of bools that are True for a given index if the key is pressed. Which the 97 is - an index to that. Did you actually bother to read the docs? http://www.pygame.org/docs/ref/key.html#pygame.key.get_pressed Diez -- http://mail.python.org/mailman/listinfo/python-list
State machine Example in Python
Can someone please redirect me for a state machine example or design pattern used in Python. Regards Alok -- http://mail.python.org/mailman/listinfo/python-list
Re: Preventing 'bad' filenames from raising errors in os.path
[EMAIL PROTECTED] wrote: Bad file names, i.e. filenames the OS considers illegal, will cause functions in the os.path module to raise an error. Example: import os.path print os.path.getsize( 'c:/pytest/*.py' ) The issue, I think, is there's no function that checks syntax of a path name without checking with the underlying OS to see if the file exists. If, say, you're writing a GUI tool for setting up some configuration, you'd want to do input validation on fields without actually accessing the files. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: #!/usr/bin/env python vs. #!/usr/bin/python
I work on an AIX system where /usr/bin and /usr/local/bin apps can only be installed by root. Our system doesn't have python or many other tools we like to use installed so we have to install python in an alternate directory location. We have a system installation of Perl installed, but it's a release or two older than what we need, so we have done the same for perl. Thus, #!/usr/bin/env whatever allows our developers to experiment without always requiring the services of the admins, who are spread too thinly amongst all the other *Nixes they have to support, and who are also separated by many layers of red tape from us. On Sat, May 3, 2008 at 10:24 AM, Lou Pecora <[EMAIL PROTECTED]> wrote: > In article <[EMAIL PROTECTED]>, > Grant Edwards <[EMAIL PROTECTED]> wrote: > > > On 2008-05-02, D'Arcy J.M. Cain <[EMAIL PROTECTED]> wrote: > > > On Sat, 03 May 2008 00:44:00 +1000 > > > Ben Finney <[EMAIL PROTECTED]<[EMAIL PROTECTED]>> > wrote: > > >> "D'Arcy J.M. Cain" <[EMAIL PROTECTED]> writes: > > >> > As someone else pointed out, not all the world is Linux. > > >> > > >> It's a good thing I've never implied such to be the case. > > > > > > You haven't *said* it but you have definitely *implied* it. > > > Installing Python in /usr/bin is not common. > > > > It is common. That's where it's installed by almost all Linux > > distributions. > > MacOS X system python (or links to them) is in the same place. > > -- > -- Lou Pecora > -- > http://mail.python.org/mailman/listinfo/python-list > -- | _ | * | _ | | _ | _ | * | | * | * | * | -- http://mail.python.org/mailman/listinfo/python-list
Re: How to use a parameter in a class
Decebal wrote: I have the following class: # class Dummy(): value = 0 def __init__(self, thisValue): print thisValue self.value = thisValue value = thisValue def testing(self): print 'In test: %d' % self.value def returnValue(self): return self.value result = someFuntion(default = value) But of course it doesn't work, as it tries to call someFunction and no such thing is define anywhere. But that can't be the answer to the *real* question you are trying to ask. Why don't you tell us what you are trying to do here, and we'll wee if we can help. Gary Herron P.S. Having a class attribute AND an instance attribute, both named "value" is going to cause trouble. # But the last line does not work. I would like to do a call like: dummy = Dummy(thisValue = 12) And that someFunction gets a default value of 12. How can I do that? -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Why is None <= 0
New style classes are classes inherited from class object. Therefore: class A: pass is oldstyle, while class B(object): pass is newstyle. On Tue, Apr 29, 2008 at 8:29 AM, blaine <[EMAIL PROTECTED]> wrote: > On Apr 29, 5:32 am, Duncan Booth <[EMAIL PROTECTED]> wrote: > > =?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?= <[EMAIL PROTECTED]> wrote: > > > (FWIW, in 2.x, x>=4?, it's None < numbers < anything else; > > > numbers are ordered by value, everything else is ordered > > > by type name, then by address, unless comparison functions > > > are implemented). > > > > Quite apart from Jon pointing out that this isn't true for all cases > when > > copmparing against None, the other half also isn't true: > > > > >>> class C: pass > > >>> C() < 5 > > > > True > > > > That happens at least in Python 2.5.2 on win32. Yet another reason to > avoid > > old-style classes. > > Sorry - but what are new style classes? > -- > http://mail.python.org/mailman/listinfo/python-list > -- | _ | * | _ | | _ | _ | * | | * | * | * | -- http://mail.python.org/mailman/listinfo/python-list
Re: How to use a parameter in a class
Hello Decebal, I am new to python myself, which might be the cause why I do not get that "last line" at all. To me it looks like you are trying to set a variable in the class body (if I am reading the indent correctly) and call a function (that does not exist?) to calculate the value for it. Does that work at all? About a week ago I learned something, that might solve your problem: You can only access the instance's variables from within a function. The variable is not visible from the outside. To get that value (12 on your example) you have to be in a function, or call your returnValue() function from the outside. I hope, that I did not waste your time with my lowest level knowledge. Have a nice weekend, Florian On Sat, May 3, 2008 at 12:05 PM, Decebal <[EMAIL PROTECTED]> wrote: > I have the following class: > # > class Dummy(): > value = 0 > def __init__(self, thisValue): > print thisValue > self.value = thisValue > value = thisValue > > def testing(self): > print 'In test: %d' % self.value > > def returnValue(self): > return self.value > > result = someFuntion(default = value) > # > > But the last line does not work. > I would like to do a call like: > dummy = Dummy(thisValue = 12) > > And that someFunction gets a default value of 12. How can I do that? > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Feature suggestion: sum() ought to use a compensated summation algorithm
I did the following calculation: Generated a list of a million random numbers between 0 and 1, constructed a new list by subtracting the mean value from each number, and then calculated the mean again. The result should be 0, but of course it will differ from 0 slightly because of rounding errors. However, I noticed that the simple Python program below gives a result of ~ 10^-14, while an equivalent Mathematica program (also using double precision) gives a result of ~ 10^-17, i.e. three orders of magnitude more precise. Here's the program (pardon my style, I'm a newbie/occasional user): from random import random data = [random() for x in xrange(100)] mean = sum(data)/len(data) print sum(x - mean for x in data)/len(data) A little research shows that Mathematica uses a "compensated summation" algorithm. Indeed, using the algorithm described at http://en.wikipedia.org/wiki/Kahan_summation_algorithm gives us a result around ~ 10^-17: def compSum(arr): s = 0.0 c = 0.0 for x in arr: y = x-c t = s+y c = (t-s) - y s = t return s mean = compSum(data)/len(data) print compSum(x - mean for x in data)/len(data) I thought that it would be very nice if the built-in sum() function used this algorithm by default. Has this been brought up before? Would this have any disadvantages (apart from a slight performance impact, but Python is a high-level language anyway ...)? Szabolcs Horvát -- http://mail.python.org/mailman/listinfo/python-list
pygame: rect moveto?
http://www.pygame.org/docs/ref/rect.html#Rect.move well i need to put my rect ina specific place and i donw know from where i will move it so i need a function rect.moveTo(x,y). how do i do that? -- http://mail.python.org/mailman/listinfo/python-list
Re: list.index crashes when the element is not found
On May 2, 3:26 pm, TkNeo <[EMAIL PROTECTED]> wrote:
> On May 2, 2:49 pm, Jeff <[EMAIL PROTECTED]> wrote:
>
> > The generally used idiom for that is:
>
> > lst = ['a', 'b', 'c']
> > if 'a' in lst:
> > foo = lst.index('a')
>
> Jeff - Gracias !!
>
> I am fairly new to python. Thanks for the example code snippet above.
> It is the same amount of code as receiving -1 and then checking for
> doing an "if else" for -1 so now i don't feel bad. But, being new to
> this paradigm, raising of an exception when it can't find the element
> appears to be weird to me for some unexplainable reason.
I don't know how Python looks up internally 'a' in the list in the
statement "if 'a' in lst: ... ," but just from common sense, I think
it would have to do the work of finding it twice, first in the
statement "'a' in lst," and then in the statement "lst.index('a')." If
you had a lot of elements in the list, and/or you were looping a lot
of times to find a lot of things, it'd be better to do the try/except
form.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Phyton module for Windows Event Viewer?
> Can someone point me in the right direction? I'm looking for a module > to monitor the Windows Event Viewer. http://docs.python.org/lib/module-logging.html NTEventLogHandler is the one you should use. happy pythoning -- http://mail.python.org/mailman/listinfo/python-list
Re: IPv6 and Python
On 3 Mag, 05:38, Roy Smith <[EMAIL PROTECTED]> wrote:
> In the application I work on, we've avoided this. We just listen on two
> separate sockets (one for each address family). We wrote a DualSocket
> class which manages the two underlying single-protocol sockets and makes
> them appear to be a single dual-protocol socket. It was a lot of user code
> to write, compared with using the mapped address mechanism, but at least
> it's portable to every OS we've seen that support IPv6.
>
> You don't need multi-threading to handle multiple sockets. In our
> implementation, for example, we use select() in a single thread to
> multiplex the two.
I would be very interested in taking a look at how you implemented
that part of code. Would it be possible?
For now I wrote this and it seems to work just fine.
Any advice about it?
class FTPServer(asyncore.dispatcher):
def __init__(self, address, handler):
"""Initiate the FTP server opening listening on address.
- (tuple) address: the ip:port pair on which to listen
for ftp data connections.
- (classobj) handler: the handler class to use.
"""
asyncore.dispatcher.__init__(self)
self.handler = handler
host, port = address
for res in socket.getaddrinfo(host, port, 0,
socket.SOCK_STREAM):
af, socktype, proto, canonname, sa = res
try:
self.create_socket(af, socket.SOCK_STREAM)
except socket.error, msg:
if self.socket:
self.socket.close()
self.socket = None
continue
break
if not self.socket:
raise socket.error, msg
self.address_family = af
if os.name not in ('nt', 'ce'):
self.set_reuse_addr()
self.bind(address)
self.listen(5)
def handle_accept(self):
sock_obj, addr = self.accept()
log("[]%s:%s Connected." %addr[0:2])
handler = self.handler(sock_obj, self)
--- Giampaolo
http://code.google.com/p/pyftpdlib/
--
http://mail.python.org/mailman/listinfo/python-list
Re: pygame: rect moveto?
On Sat, 03 May 2008 09:51:06 -0700, globalrev wrote: > http://www.pygame.org/docs/ref/rect.html#Rect.move > > well i need to put my rect ina specific place and i donw know from > where i will move it so i need a function rect.moveTo(x,y). > > how do i do that? No need for a function or method:: rect.topleft = (x, y) Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
Szabolcs Horvát <[EMAIL PROTECTED]> writes: [...] > A little research shows that Mathematica uses a "compensated > summation" algorithm. Indeed, using the algorithm described at > http://en.wikipedia.org/wiki/Kahan_summation_algorithm > gives us a result around ~ 10^-17: > > def compSum(arr): > s = 0.0 > c = 0.0 > for x in arr: > y = x-c > t = s+y > c = (t-s) - y > s = t > return s > > mean = compSum(data)/len(data) > print compSum(x - mean for x in data)/len(data) > > > I thought that it would be very nice if the built-in sum() function > used this algorithm by default. Has this been brought up before? > Would this have any disadvantages (apart from a slight performance > impact, but Python is a high-level language anyway ...)? > > Szabolcs Horvát sum() works for any sequence of objects with an __add__ method, not just floats! Your algorithm is specific to floats. -- Arnaud -- http://mail.python.org/mailman/listinfo/python-list
Re: Getting started with pyvtk
On Fri, 02 May 2008 17:40:02 +0200, Paul Melis wrote: > > I'm not sure you've been helped so far as you seem to already understand > about pyvtk not being the official VTK bindings :) > > So, what would you like to know? Thanks, I think I'm set. For the benefit of the next instance of me googling around trying to get started driving VTK from Python, here's what I wanted to find: - Ignore pyvtk. - Import vtk. - To see how to drive VTK from Python, run all the files of the form /usr/share/vtk/.../*.py and see which ones do something like what you want to do. -- To email me, substitute nowhere->spamcop, invalid->net. -- http://mail.python.org/mailman/listinfo/python-list
Re: simple beginner question about lists and negative index
On May 1, 10:04 am, "Terry Reedy" <[EMAIL PROTECTED]> wrote: > "jmDesktop" <[EMAIL PROTECTED]> wrote in message > | > | s = 'abcde' > | i = -1 > | for i in range (-1, -len(s), -1): > |print s[:i], i > | Why doesn't the first one have the e if -1 is the end of the list? That should be obvious. 'print s[-1]' will yield the "e", but your code is using slicing, s[:i], not printing elements from a list. Your code starts with the value s[:-1], which is exactly the same as s[0:-1], so you print the string from 0 to -1 which omits the last letter. The next time it omits the last 2 letters s[:-2]. As already stated, that should be very obvious. Also, if you want a response in the future, make your code readable. That means not using, "i", "l", or "o" as variable names. They look too much like numbers and some people won't waste their time trying to distinguish between them. There are 23 other letters that work just a well. -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
On Sat, 03 May 2008 18:50:34 +0200, Szabolcs Horvát wrote: > I did the following calculation: Generated a list of a million random > numbers between 0 and 1, constructed a new list by subtracting the mean > value from each number, and then calculated the mean again. > > The result should be 0, but of course it will differ from 0 slightly > because of rounding errors. > > However, I noticed that the simple Python program below gives a result > of ~ 10^-14, while an equivalent Mathematica program (also using double > precision) gives a result of ~ 10^-17, i.e. three orders of magnitude > more precise. > > Here's the program (pardon my style, I'm a newbie/occasional user): > > from random import random > > data = [random() for x in xrange(100)] > > mean = sum(data)/len(data) > print sum(x - mean for x in data)/len(data) > > A little research shows that Mathematica uses a "compensated summation" > algorithm. Indeed, using the algorithm described at > http://en.wikipedia.org/wiki/Kahan_summation_algorithm gives us a result > around ~ 10^-17: > > def compSum(arr): > s = 0.0 > c = 0.0 > for x in arr: > y = x-c > t = s+y > c = (t-s) - y > s = t > return s > > mean = compSum(data)/len(data) > print compSum(x - mean for x in data)/len(data) > > > I thought that it would be very nice if the built-in sum() function used > this algorithm by default. Has this been brought up before? Would this > have any disadvantages (apart from a slight performance impact, but > Python is a high-level language anyway ...)? > > Szabolcs Horvát Built-in sum should work with everything, not just floats. I think it would be useful addition to standard math module. If you know C you could write a patch to mathmodule.c and submit it to Python devs. -- Ivan -- http://mail.python.org/mailman/listinfo/python-list
Re: Problems replacing \ with \\
Sorry for the late response. I've been travelling internationally and am just getting back to work. So--thank you to everyone who responded! To answer everyone's question I am dumping all of the data from a mysql database, then creating a postgresql database, then inserting the data into the new postgresql database. This seems to be a reasonable way to do it, because all I have to do is double my 's and my \s (I hope). Thanks On Apr 21, 7:12 pm, Michael Torrie <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: > > Hi... > > > Here's a weird problem...I'm trying to escape a bunch of data to put > > into a database. > > Is it possible to use the database API and prepared statements to avoid > having to go through this exercise? Also, most database APIs work > natively in unicode, so creating your prepared statement and then > passing in each parameter as an argument saves a lot of hassle, since > the API's usually decode the unicode strings automatically. -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
Szabolcs Horvát <[EMAIL PROTECTED]> wrote: > I did the following calculation: Generated a list of a million random > numbers between 0 and 1, constructed a new list by subtracting the mean > value from each number, and then calculated the mean again. > > The result should be 0, but of course it will differ from 0 slightly > because of rounding errors. > > However, I noticed that the simple Python program below gives a result > of ~ 10^-14, while an equivalent Mathematica program (also using double > precision) gives a result of ~ 10^-17, i.e. three orders of magnitude > more precise. > > Here's the program (pardon my style, I'm a newbie/occasional user): > > from random import random > > data = [random() for x in xrange(100)] > > mean = sum(data)/len(data) > print sum(x - mean for x in data)/len(data) > > A little research shows that Mathematica uses a "compensated summation" > algorithm. Indeed, using the algorithm described at > http://en.wikipedia.org/wiki/Kahan_summation_algorithm > gives us a result around ~ 10^-17: I was taught in my numerical methods course (about 20 years ago now!) that the way to do this sum with most accuracy is to sum from the smallest magnitude to the largest magnitude. Eg >>> from random import random >>> data = [random() for x in xrange(100)] >>> mean = sum(data)/len(data) >>> print sum(x - mean for x in data)/len(data) 1.64152134108e-14 >>> mean = sum(sorted(data))/len(data) >>> print sum(x - mean for x in data)/len(data) 5.86725534824e-15 >>> It isn't as good as the compensated sum but it is easy! > I thought that it would be very nice if the built-in sum() function used > this algorithm by default. Has this been brought up before? Would this > have any disadvantages (apart from a slight performance impact, but > Python is a high-level language anyway ...)? sum() gets used for any numerical types not just floats... -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
Arnaud Delobelle wrote: sum() works for any sequence of objects with an __add__ method, not just floats! Your algorithm is specific to floats. This occurred to me also, but then I tried sum(['abc', 'efg'], '') and it did not work. Or is this just a special exception to prevent the misuse of sum to join strings? (As I said, I'm only an occasional user.) Generally, sum() seems to be most useful for numeric types (i.e. those that form a group with respect to __add__ and __neg__/__sub__), which may be either exact (e.g. integers) or inexact (e.g. floating point types). For exact types it does not make sense to use compensated summation (though it wouldn't give an incorrect answer, either), and sum() cannot decide whether a user-defined type is exact or inexact. But I guess that it would still be possible to make sum() use compensated summation for built-in floating point types (float/complex). Or, to go further, provide a switch to choose between the two methods, and make use compensated summation for float/complex by default. (But perhaps people would consider this last alternative a bit too messy.) (Just some thoughts ...) -- http://mail.python.org/mailman/listinfo/python-list
Re: Python and SOAP status
Jeff wrote: Twisted has SOAP support. yes but it is based on no longer actively maintained SOAPpy. -- http://mail.python.org/mailman/listinfo/python-list
Light slices + COW
Sometimes different languages suggests me ways to cross-pollinate
them.
(Note: probably someone has already suggested (and even implemented)
the following ideas, but I like to know why they aren't fit for
Python).
Python generators now allow me to package some code patterns common in
my code, that other (older) languages didn't allow me to extract; for
example given a sequence (array, list, etc) of items now and then I
need to scan all the upper triangle of their cross matrix:
def xpairs(seq):
len_seq = len(seq)
for i, e1 in enumerate(seq):
for j in xrange(i+1, len_seq):
yield e1, seq[j]
Or adjacent ones (there are ways to generalize the following function,
but this the most common situation for me):
def xpairwise(iterable):
return izip(iterable, islice(iterable, 1, None))
That xpairs() generator is nice, but it's not the best possible code
(but you may disagree with me, and you may think that code better than
the successive D code that uses two slices). Inside it I can't use a
list slicing because it copies subparts of the list, probably becoming
too much slow (for example if len(seq) == 1000).
Compared to Python, the D language is at lower level, so you may
expect it to have a worse syntax (but the ShedSkin compiler has shown
me once for all that you can have a language that is both high-level,
with a short & sexy syntax, and very fast), but you can define a
xpairs() iterable class in it too, and the core of that iterable class
there's a method that may look like this:
if (this.seq.length > 1)
foreach (i, e1; this.seq[0 .. $-1])
foreach (e2; this.seq[i+1 .. $]) {
result = dg(e1, e2); if (result) break; // yield
}
Where the strange last line is the yield, and the $ inside [] is the
length of the current array (a very clever idea that avoids the need
for negative indexes).
That D code is as fast or faster than the code you can back-translate
from Python, this is possible because in D arrays slices are very
light, they are a struct of (in the future they may
change to a couple of pointers, to speed up the array slice scanning).
So if you slice an array you don't copy its contents, just the start-
end points of the slice. If you read/scan the slice that's all you
have, while if you write on it, D uses a Copy-On-Write strategy, that
is a just-in-time copy of the slice contents.
In practice this speeds up lot of code that manages strings and arrays
(like a XML parser, making it among the faster ones).
Being the slice a struct, it's stack allocated, so the inner foreach
doesn't even create a slice object in the heap each time the outer
foreach loops.
One problem with this strategy is that if you have a huge string, and
you keep only a little slice of it, the D garbage collector will keep
it all in memory. To avoid that problem you have to duplicate the
slice manually:
somestring[inf...sup].dup;
I think Psyco in some situations is able to manage string slices
avoiding the copy.
I think Python 3.0 too may enjoy a similar strategy of light slices +
COW for strings, lists and arrays (tuples and strings don't need the
COW).
Bye,
bearophile
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python and SOAP status
Heikki Toivonen wrote: I have started researching the Python SOAP (and web services in general) options out there. Python 2.5 should be supported. I used Python for some web services stuff (demo quality) a few years back without major problems. However, it seems many of the libraries I remember from that time have become dormant or have been explicitly discontinued. A colleague also commented that he run into lots of problems trying to use Python for SOAP a year ago. What is your experience with Python SOAP, WSDL etc. libraries? What are the good, maintained options out there right now? How is standards compliance, how robust are the libraries, how do they interoperate with other libraries written in other languages (I am especially interested in interoperability with Java and PHP web services stacks). It seems like the top 3 candidates on my list are ZSI (http://pywebsvcs.sourceforge.net/), soaplib (http://trac.optio.webfactional.com/) and suds (http://trac.optio.webfactional.com/). If you have any experience in using these, I'd very much like to hear from you. There was quite a depressing post about ZSI's status at http://www.kunxi.org/archives/2008/01/pythonsoap-second-encounter/. I had good experience using soaplib as the server and Java client. For the client side I found elementsoap (http://effbot.org/downloads/) to be useful on a couple occasions. Waldemar -- http://mail.python.org/mailman/listinfo/python-list
Re: State machine Example in Python
On Sat, May 3, 2008 at 6:26 PM, Alok Kumar <[EMAIL PROTECTED]> wrote: > Can someone please redirect me for a state machine example or design pattern > used in Python. > Google for it, eg: http://www.google.com/search?q=python+state+machine There's plenty of results. Be more specific if you can't get what you want from Google. David. -- http://mail.python.org/mailman/listinfo/python-list
Re: State machine Example in Python
> There's plenty of results. Be more specific if you can't get what you > want from Google. You'll probably find what you're looking for here: http://wiki.python.org/moin/FiniteStateMachine David. -- http://mail.python.org/mailman/listinfo/python-list
Re: Do you know of a much simpler way of writing a program that writes a program?
In article <[EMAIL PROTECTED]>, Jeff <[EMAIL PROTECTED]> wrote: > >Use lisp? :-))) -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ Help a hearing-impaired person: http://rule6.info/hearing.html -- http://mail.python.org/mailman/listinfo/python-list
Re: #!/usr/bin/env python vs. #!/usr/bin/python
In article <[EMAIL PROTECTED]>, Gabriel Genellina <[EMAIL PROTECTED]> wrote: > >I can't believe some angry responses in this thread - it's just a >technical question, not about which is the best team in the [preferred >sports here] National Championship... http://www.netfunny.com/rhf/jokes/91q3/usolym.html -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ Help a hearing-impaired person: http://rule6.info/hearing.html -- http://mail.python.org/mailman/listinfo/python-list
Re: IPv6 and Python
In article "Giampaolo Rodola'" <[EMAIL PROTECTED]> wrote: > I would be very interested in taking a look at how you implemented > that part of code. Would it be possible? Sadly, no. It's proprietary code. But, it's really not that complicated to reconstruct, once you know the basic idea. You have a DualSocket class whose constructor creates two sockets. It exposes (roughly) the same interface as a socket object, but decides which underlying protocol-specific socket to delegate to based on the address family of the sockaddr you give it. Oh, yeah, this was done in C++, but there's nothing really that's language specific in the implementation. -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
On May 3, 3:44 pm, Szabolcs Horvát <[EMAIL PROTECTED]> wrote: > Arnaud Delobelle wrote: > > > sum() works for any sequence of objects with an __add__ method, not > > just floats! Your algorithm is specific to floats. > > This occurred to me also, but then I tried > > sum(['abc', 'efg'], '') > > and it did not work. Or is this just a special exception to prevent the > misuse of sum to join strings? (As I said, I'm only an occasional user.) > > Generally, sum() seems to be most useful for numeric types (i.e. those > that form a group with respect to __add__ and __neg__/__sub__), which > may be either exact (e.g. integers) or inexact (e.g. floating point > types). For exact types it does not make sense to use compensated > summation (though it wouldn't give an incorrect answer, either), and > sum() cannot decide whether a user-defined type is exact or inexact. > > But I guess that it would still be possible to make sum() use > compensated summation for built-in floating point types (float/complex). > > Or, to go further, provide a switch to choose between the two methods, > and make use compensated summation for float/complex by default. (But > perhaps people would consider this last alternative a bit too messy.) > > (Just some thoughts ...) The benefits should be weighted considering the common case. For example, do you find an error of 10^-14 unbearable ? How many times someone will add 1.000.000 numbers in a typical scientific application written in python ? I believe that moving this to third party could be better. What about numpy ? Doesn't it already have something similar ? -- http://mail.python.org/mailman/listinfo/python-list
dict invert - learning question
Hello,
here is a piece of code I wrote to check the frequency of values and
switch them around to keys in a new dictionary. Just to measure how
many times a certain key occurs:
def invert(d):
inv = {}
for key in d:
val = d[key]
if val not in inv:
inv.setdefault(val, [key])
else:
inv[val].append(key)
return inv
Using the methods above (I'm just a beginner) could I have written it
more concisely? Any criticism/critique in the code would be greatly
appreciated.
Many Thanks,
dave
--
http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
Szabolcs Horvát <[EMAIL PROTECTED]> writes: > Arnaud Delobelle wrote: >> >> sum() works for any sequence of objects with an __add__ method, not >> just floats! Your algorithm is specific to floats. > > This occurred to me also, but then I tried > > sum(['abc', 'efg'], '') > > and it did not work. Or is this just a special exception to prevent > the misuse of sum to join strings? (As I said, I'm only an occasional > user.) I think that's right: anything with an __add__ method, apart from string, can be sum()ed. -- Arnaud -- http://mail.python.org/mailman/listinfo/python-list
Daily WTF with XML, or error handling in SAX
So I set out to learn handling three-letter-acronym files in Python,
and SAX worked nicely until I encountered badly formed XMLs, like with
bad characters in it (well Unicode supposed to handle it all but
apparently doesn't), using http://dchublist.com/hublist.xml.bz2 as
example data, with goal to extract Users and Address properties where
number of Users is greater than given number.
So I extended my First XML Example with an error handler:
# = snip ===
from xml.sax import make_parser
from xml.sax.handler import ContentHandler
from xml.sax.handler import ErrorHandler
class HubHandler(ContentHandler):
def __init__(self, hublist):
self.Address = ''
self.Users = ''
hl = hublist
def startElement(self, name, attrs):
self.Address = attrs.get('Address',"")
self.Users = attrs.get('Users', "")
def endElement(self, name):
if name == "Hub" and int(self.Users) > 2000:
#print self.Address, self.Users
hl.append({self.Address: int(self.Users)})
class HubErrorHandler(ErrorHandler):
def __init__(self):
pass
def error(self, exception):
import sys
print "Error, exception: %s\n" % exception
def fatalError(self, exception):
print "Fatal Error, exception: %s\n" % exception
hl = []
parser = make_parser()
hHandler = HubHandler(hl)
errHandler = HubErrorHandler()
parser.setContentHandler(hHandler)
parser.setErrorHandler(errHandler)
fh = file('hublist.xml')
parser.parse(fh)
def compare(x,y):
if x.values()[0] > y.values()[0]:
return 1
elif x.values()[0] < y.values()[0]:
return -1
return 0
hl.sort(cmp=compare, reverse=True)
for h in hl:
print h.keys()[0], " ", h.values()[0]
# = snip ===
And then BAM, Pythonwin has hit me:
>>> execfile('ph.py')
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
>>> RESTART
Just before the "RESTART" line, Windows has announced it killed
pythonw.exe process (I suppose it was a child process).
WTF is happening here? Wasn't fatalError method in the HubErrorHandler
supposed to handle the invalid tokens? And why is the message repeated
many times? My method is called apparently, but something in SAX goes
awry and the interpreter crashes.
--
http://mail.python.org/mailman/listinfo/python-list
Re: unified command line args, environment variables, .conf file settings.
On May 2, 11:29 pm, Ben Finney <[EMAIL PROTECTED]>
wrote:
> smitty1e <[EMAIL PROTECTED]> writes:
> > Just a fun exercise to unify some of the major input methods for a
> > script into a single dictionary.
> > Here is the output, given a gr.conf file in the same directory with
> > the contents stated below:
>
> > [EMAIL PROTECTED] ~/proj/mddl4/test $ ./inputs.py
> > {'source_db': '/home/sweet/home.db'}
> > [EMAIL PROTECTED] ~/proj/mddl4/test $ source_db="test_env" ./inputs.py
> > {'source_db': 'test_env'}
> > [EMAIL PROTECTED] ~/proj/mddl4/test $ ./inputs.py -f "test_cli"
> > {'source_db': 'test_cli'}
>
> A good start. However, you need to account for two conventions with
> configuration of programs via environment variables:
[snip]
> This is probably best done by a mapping specifically for the
> environment variable names:
>
> config_env_names = {
> 'source_db': 'GR_SOURCE_DB',
> }
>
> and then referencing that dictionary when inspecting the environment:
>
> for key in config.keys():
> # ...
> environ_name = config_env_names[key]
> environ_value = os.environ.get(environ_name)
> if environ_value is not None:
> config[key] = environ_value
> #...
Hmmm...
I kinda think your excellent points are more in the realm of policy
than mechanism.
What I was going for was to assert that there a designated section (of
many possible ones)
in a .conf file that represents the set of possible program inputs,
and then
offering an example of letting environment and command line arguments
override that
.conf section.
Your points for reducing collision are well taken. Possibly having a
removable prefix
(which could be a "magic" .conf entry) would allow for a de-conflictor
prefix like
MY_PROGRAM_REALLY_ROCKSsource_db without having to propagate such an
eyesore throughout
the program code.
In any case, it stands as a fun little worked example.
R,
C
--
http://mail.python.org/mailman/listinfo/python-list
Re: dict invert - learning question
dave <[EMAIL PROTECTED]> wrote:
> Hello,
>
> here is a piece of code I wrote to check the frequency of values and
> switch them around to keys in a new dictionary. Just to measure how
> many times a certain key occurs:
>
> def invert(d):
> inv = {}
> for key in d:
> val = d[key]
> if val not in inv:
>inv.setdefault(val, [key])
> else:
>inv[val].append(key)
> return inv
>
>
> Using the methods above (I'm just a beginner) could I have written it
> more concisely? Any criticism/critique in the code would be greatly
> appreciated.
>
The obvious change I'd make is:
if val not in inv:
inv[val] = key
else:
... etc.
The setdefault method is only appropriate if you don't know
whether or not the key is in the dictionary. since you've already tests
that you should just go ahead and set the value.
Otherwise it all seems fine.
--
http://mail.python.org/mailman/listinfo/python-list
Re: dict invert - learning question
dave <[EMAIL PROTECTED]> writes:
> Hello,
>
> here is a piece of code I wrote to check the frequency of values and
> switch them around to keys in a new dictionary. Just to measure how
> many times a certain key occurs:
>
> def invert(d):
> inv = {}
> for key in d:
> val = d[key]
> if val not in inv:
> inv.setdefault(val, [key])
You can simply write:
inv[val] = [key]
> else:
> inv[val].append(key)
> return inv
>
>
> Using the methods above (I'm just a beginner) could I have written it
> more concisely? Any criticism/critique in the code would be greatly
> appreciated.
Apart from the unnecessary use of setdefault, it looks good to me.
* You could change if 'val not in inv:' to 'if val in inv:' (and swap
the if and else clauses of course) in order to have a positive
condition rather than a negative one
* If you want to use setdefault, you can replace the if .. else
construct by:
inv.setdefault(val, []).append(key)
* You can also iterate over keys and values using the items() or
iteritems() method of dictionaries:
def invert(d):
inv = {}
for key, val in d.iteritems():
inv.setdefault(val, []).append(key)
return inv
--
Arnaud
--
http://mail.python.org/mailman/listinfo/python-list
Error handling in SAX
(this is a repost, for it's been a while since I posted this text via
Google Groups and it plain didn't appear on c.l.py - if it did appear
anyway, apols)
So I set out to learn handling three-letter-acronym files in Python,
and SAX worked nicely until I encountered badly formed XMLs, like with
bad characters in it (well Unicode supposed to handle it all but
apparently doesn't), using http://dchublist.com/hublist.xml.bz2 as
example data, with goal to extract Users and Address properties where
number of Users is greater than given number.
So I extended my First XML Example with an error handler:
# = snip ===
from xml.sax import make_parser
from xml.sax.handler import ContentHandler
from xml.sax.handler import ErrorHandler
class HubHandler(ContentHandler):
def __init__(self, hublist):
self.Address = ''
self.Users = ''
hl = hublist
def startElement(self, name, attrs):
self.Address = attrs.get('Address',"")
self.Users = attrs.get('Users', "")
def endElement(self, name):
if name == "Hub" and int(self.Users) > 2000:
#print self.Address, self.Users
hl.append({self.Address: int(self.Users)})
class HubErrorHandler(ErrorHandler):
def __init__(self):
pass
def error(self, exception):
import sys
print "Error, exception: %s\n" % exception
def fatalError(self, exception):
print "Fatal Error, exception: %s\n" % exception
hl = []
parser = make_parser()
hHandler = HubHandler(hl)
errHandler = HubErrorHandler()
parser.setContentHandler(hHandler)
parser.setErrorHandler(errHandler)
fh = file('hublist.xml')
parser.parse(fh)
def compare(x,y):
if x.values()[0] > y.values()[0]:
return 1
elif x.values()[0] < y.values()[0]:
return -1
return 0
hl.sort(cmp=compare, reverse=True)
for h in hl:
print h.keys()[0], " ", h.values()[0]
# = snip ===
And then BAM, Pythonwin has hit me:
>>> execfile('ph.py')
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
Fatal Error, exception: hublist.xml:2247:11: not well-formed (invalid
token)
>>> RESTART
Just before the "RESTART" line, Windows has announced it killed
pythonw.exe process (I suppose it was a child process).
WTF is happening here? Wasn't fatalError method in the HubErrorHandler
supposed to handle the invalid tokens? And why is the message repeated
many times? My method is called apparently, but something in SAX goes
awry and the interpreter crashes.
--
http://mail.python.org/mailman/listinfo/python-list
Re: dict invert - learning question
thanks Duncan and Arnaud.
I'm learning Python from the "How to Think Like a Python Programmer"
book by Allen Downey. My first try used the "inv[val] = [key]" and
then the next problem was to incorporate the "D.setdefault(...)" method.
Thank you for your help. I'm always amazed how kind people are in this group.
On 2008-05-03 14:57:29 -0600, Arnaud Delobelle <[EMAIL PROTECTED]> said:
dave <[EMAIL PROTECTED]> writes:
Hello,
here is a piece of code I wrote to check the frequency of values and
switch them around to keys in a new dictionary. Just to measure how
many times a certain key occurs:
def invert(d):
inv = {}
for key in d:
val = d[key]
if val not in inv:
inv.setdefault(val, [key])
You can simply write:
inv[val] = [key]
else:
inv[val].append(key)
return inv
Using the methods above (I'm just a beginner) could I have written it
more concisely? Any criticism/critique in the code would be greatly
appreciated.
Apart from the unnecessary use of setdefault, it looks good to me.
* You could change if 'val not in inv:' to 'if val in inv:' (and swap
the if and else clauses of course) in order to have a positive
condition rather than a negative one
* If you want to use setdefault, you can replace the if .. else
construct by:
inv.setdefault(val, []).append(key)
* You can also iterate over keys and values using the items() or
iteritems() method of dictionaries:
def invert(d):
inv = {}
for key, val in d.iteritems():
inv.setdefault(val, []).append(key)
return inv
--
http://mail.python.org/mailman/listinfo/python-list
Re: list.index crashes when the element is not found
Jeff wrote:
The generally used idiom for that is:
lst = ['a', 'b', 'c']
if 'a' in lst:
foo = lst.index('a')
It's not a very good idiom, since it iterates over the list twice
unnecessarily: first, to see if the object is in the list; then, to
find the index of that object. That's pointless wasteful.
The Pythonic idiom is to catch the exception and then deal with it as
desired.
--
Erik Max Francis && [EMAIL PROTECTED] && http://www.alcyone.com/max/
San Jose, CA, USA && 37 18 N 121 57 W && AIM, Y!M erikmaxfrancis
--
http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
Szabolcs Horvát wrote: Arnaud Delobelle wrote: sum() works for any sequence of objects with an __add__ method, not just floats! Your algorithm is specific to floats. This occurred to me also, but then I tried sum(['abc', 'efg'], '') and it did not work. Or is this just a special exception to prevent the misuse of sum to join strings? (As I said, I'm only an occasional user.) What you wrote is nonsensical there, no different from 'a' + 1 -- which is why it quite rightly raises a TypeError. You're trying to add a list to a string, which is nonsensical. You add strings to strings, or lists to lists, but mixing them up doesn't make sense. Python can't guess what you mean when you write something like ['abc', 'def'] + '' -- which is the functional equivalent of your call to sum -- and so doesn't try. It indicates this by raising a TypeError. -- Erik Max Francis && [EMAIL PROTECTED] && http://www.alcyone.com/max/ San Jose, CA, USA && 37 18 N 121 57 W && AIM, Y!M erikmaxfrancis -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
Hallöchen! Erik Max Francis writes: > Szabolcs Horvát wrote: > >> Arnaud Delobelle wrote: >>> >>> sum() works for any sequence of objects with an __add__ method, not >>> just floats! Your algorithm is specific to floats. >> >> This occurred to me also, but then I tried >> >> sum(['abc', 'efg'], '') >> >> and it did not work. Or is this just a special exception to >> prevent the misuse of sum to join strings? (As I said, I'm only >> an occasional user.) > > What you wrote is nonsensical there, no different from 'a' + 1 -- > which is why it quite rightly raises a TypeError. No, the above expression should yield ''+'abc'+'efg', look for the signature of sum in the docs. Tschö, Torsten. -- Torsten Bronger, aquisgrana, europa vetus Jabber ID: [EMAIL PROTECTED] (See http://ime.webhop.org for further contact info.) -- http://mail.python.org/mailman/listinfo/python-list
using sqlite3 - execute vs. executemany; committing ...
Hi all,
I'd like to ask about some (probably elementary) things about the proper
usage of sqlite3 in python (2.5.2; win).
- Are there any peculiarities with using curs.executemany(...) vs. multiple
curs.execute(...) ? I read a notice, sqlite3 does internally some
caching, hence both should be similarly fast, but in my case executemany(...)
is quite a bit faster (probably due to the function call overhead?). Are
there maybe some caveats with using executemany? (the obvious
memory consumption for the intermediate list doesn't seem to matter
much in my application).
Further, I am not quite sure about the standard usage of the cursor
object and also the proper commiting the transactions and closing the
connection.
Should one create a cursor of a connection and call the execute ... methods
of the cursor -
or is it better to call the shortcut execute etc. methods of the
Connection object
directly (as suggested in the docs:
http://docs.python.org/lib/node351.html(or are there specific use
cases for both approaches)?
When the transactions should be
commited? (creating, altering a table, or also selecting the results ?)
There seem to be some implicit handling of the transactions (
http://docs.python.org/lib/sqlite3-Controlling-Transactions.html#sqlite3-Controlling-Transactions);
hence I am not
sure about the standard usage of these methods; the same is true of
connection.close() - or are these calls eventually unnecessary?
To give a bit more info about my use case, my (gui) app displays a set texts
along with some informations on the currently visible text segments.
After starting a gui the texts are read from the source files (also
containing a kind of tags) - simultaneously
the tags are parsed and the corresponding values are stored in sqlite
"in memory" - the column names as well as the corresponding values for given
text parts are extracted dynamically from the source files.
During the whole run of the program the db should be available in order to
retrieve the properties of the given text index.
Below is a part of my code dealing with the populating of the db;, I'm
sorry, it's a not runable pseudocode, however, my present code works
somehow, but I'm quite sure, the implementation isn't really standard, hence
I'd like to hear some comments/suggestions. (I asked some time ago about the
possibility of a parametrised input of the table and column names, but this
seems impossible - therefore I used the string interpolation.)
- the variables are defined in other parts of code before; most of
the names might be self explanatory; self - custom class managing the text
with its properties; text_name - name of the table; index_col_name - a name
of the special db column containing the text index; act_db_columns - list of
the db columns; question_marks - a string "?, ?, ? ..." (synchronized with
the length of act_db_columns); tags_seq - a nested list with
the previously retrieved data for the database.
conn_tags_DB = sqlite3.connect(':memory:')
curs = self.conn_tags_DB.cursor()
curs.execute('CREATE TABLE IF NOT EXISTS "%s" ("%s", UNIQUE("%s"))' %
(self.text_name, index_col_name, index_col_name))
curs.execute(u'INSERT OR REPLACE INTO "%s"("%s") VALUES (?)' %
(self.text_name, index_col_name), (0,))
for new_col in act_db_columns[1:]: # adds the needed columns (except of the
first one: index_col_name)
curs.execute('ALTER TABLE "%s" ADD "%s" TEXT' % (self.text_name,
new_col))
curs.executemany('INSERT OR REPLACE INTO "%s" VALUES (%s)' %
(self.text_name, question_marks), tags_seq)
self.conn_tags_DB.commit()
Are there maybe any comments or hints on a more elegant/efficient solution?
Now, what's
the usual way to access the database? Is it possible/wise/standard ...
to leave the connection open for the subsequent queries
during the whole run of the app; could even the
cursor eventually be present as a class method, or should it rather be
created repeatedly with each call? (After populating, the db shouldn't be
modified, but only read.)
Many thanks in advance for your help;
and apologies for this longer post.
Greetings,
Vlasta
--
http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
Torsten Bronger wrote: No, the above expression should yield ''+'abc'+'efg', look for the signature of sum in the docs. You're absolutely right, I misread it. Sorry about that. -- Erik Max Francis && [EMAIL PROTECTED] && http://www.alcyone.com/max/ San Jose, CA, USA && 37 18 N 121 57 W && AIM, Y!M erikmaxfrancis -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
On Sat, 03 May 2008 20:44:19 +0200, Szabolcs Horvát wrote: > Arnaud Delobelle wrote: >> >> sum() works for any sequence of objects with an __add__ method, not >> just floats! Your algorithm is specific to floats. > > This occurred to me also, but then I tried > > sum(['abc', 'efg'], '') Interesting, I always thought that sum is like shortcut of reduce(operator.add, ...), but I was mistaken. reduce() is more forgiving: reduce(operator.add, ['abc', 'efg'], '' ) # it works 'abcefg' -- Ivan -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
On May 3, 10:13 pm, hdante <[EMAIL PROTECTED]> wrote: > I believe that moving this to third party could be better. What about > numpy ? Doesn't it already have something similar ? Yes, Kahan summation makes sence for numpy arrays. But the problem with this algorithm is optimizing compilers. The programmer will be forced to use tricks like inline assembly to get around the optimizer. If not, the optimizer would remove the desired features of the algorithm. But then we have a serious portability problem. -- http://mail.python.org/mailman/listinfo/python-list
PIL JPEG mis-step
I got the Python Imaging Library from source, built and installed, on Ubuntu 7.10, not realizing I could run a self-test first. libjpeg is on the machine, but was not detected.. so no JPG encoder. I got the dev-libjpg and rebuilt PIL. The self-test now shows JPG support. but running setup.py install again seems to succeed, but still no JPG :-( fix ideas appreciated -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
On Sat, 2008-05-03 at 21:37 +, Ivan Illarionov wrote: > On Sat, 03 May 2008 20:44:19 +0200, Szabolcs Horvát wrote: > > > Arnaud Delobelle wrote: > >> > >> sum() works for any sequence of objects with an __add__ method, not > >> just floats! Your algorithm is specific to floats. > > > > This occurred to me also, but then I tried > > > > sum(['abc', 'efg'], '') > > Interesting, I always thought that sum is like shortcut of > reduce(operator.add, ...), but I was mistaken. > > reduce() is more forgiving: > reduce(operator.add, ['abc', 'efg'], '' ) # it works > 'abcefg' Hm, it works for lists: sum(([1], [2]), []) [1, 2] However I find the seccond argument hack ugly. Does the sum way have any performance advantages over the reduce way? -- Best Regards, Med Venlig Hilsen, Thomas -- http://mail.python.org/mailman/listinfo/python-list
Re: Problems with psycopg2
The thing is this query works fine on the console through psql, but not in my code? can anyone explain me why? On Thu, May 1, 2008 at 9:31 PM, David Anderson <[EMAIL PROTECTED]> wrote: > Hi all > I have this function: > def checkName(self, name): > cur = self.conn.cursor() > > sql = "SELECT * from patient WHERE fn_pat = '" + name + "'" > cur.execute(sql) > rows = cur.fetchall() > > if rows == "[]": > self.insert() > > It seems to work fine, But I'm getting this exception: > psycopg2.ProgrammingError: current transaction is aborted, commands > ignored until end of transaction block > at: cur.execute(sql) > > What's the problem? > thx > > ps: fn_pat is the column of the db, name is the string passed in the > function parameter > -- http://mail.python.org/mailman/listinfo/python-list
Encoding Text
Hi, I'm learning this and I'm making a program which takes RSS feeds and processes them and then outputs them to a HTML file. The problem I have is that some of the RSS feeds contain chachters which I think are outside of the ascii range as when I attempt to write the file containing themI get the follwoign error: File "C:\Users\Paul\Desktop\python\Feed reader\test.py", line 201, in runner(1,"save.ahc") File "C:\Users\Paul\Desktop\python\Feed reader\test.py", line 185, in runner h.write(html[countera]) UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 1147: ordinal not in range(128) For the life of me I can't figure out what I should do to stop this - I tried to change the html by doing a html[countera] = unicode(html[countera]) but it didn't seem to chaneg anything. Any help would be greatly appreciated, Thanks Paul > -- http://mail.python.org/mailman/listinfo/python-list
Re: PIL JPEG mis-step
On Sat, 03 May 2008 17:01:44 -0700, darkblueB wrote: > On May 3, 4:52 pm, Ivan Illarionov <[EMAIL PROTECTED]> wrote: >> Try run 'python setup.py build_ext -f' to force setup.py to rebuild >> everything with JPEG. And 'sudo python setup.py install' should install >> PIL with JPEG support. > > yes, that works > > (the self test still gives misleading results ?) but running some sample > program works fine > > thanks You're welcome. To make self test inside source directory work you probably should rebuild it inplace too. python setup.py build_ext -f -i -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
On May 3, 7:12 pm, Ivan Illarionov <[EMAIL PROTECTED]> wrote: > On Sun, 04 May 2008 00:31:01 +0200, Thomas Dybdahl Ahle wrote: > > On Sat, 2008-05-03 at 21:37 +, Ivan Illarionov wrote: > >> On Sat, 03 May 2008 20:44:19 +0200, Szabolcs Horvát wrote: > > >> > Arnaud Delobelle wrote: > > >> >> sum() works for any sequence of objects with an __add__ method, not > >> >> just floats! Your algorithm is specific to floats. > > >> > This occurred to me also, but then I tried > > >> > sum(['abc', 'efg'], '') > > >> Interesting, I always thought that sum is like shortcut of > >> reduce(operator.add, ...), but I was mistaken. > > >> reduce() is more forgiving: > >> reduce(operator.add, ['abc', 'efg'], '' ) # it works 'abcefg' > > Hm, it works for lists: > > sum(([1], [2]), []) > > [1, 2] So it's not reduce() that is more forgiving, it's sum() that makes an exception for strings only. > > However I find the seccond argument hack ugly. Does the sum way have any > > performance advantages over the reduce way? > > Yes, sum() is faster: > > $ python -m timeit "" "sum([[1], [2], [3, 4]], [])" > 10 loops, best of 3: 6.16 usec per loop > > $ python -m timeit "import operator" \ > "reduce(operator.add, [[1], [2], [3, 4]])" > 10 loops, best of 3: 11.9 usec per loop Not really; you measure the import and the attribute access time in the second case. sum() is 2x faster for adding numbers only: # Adding integers python -mtimeit --setup="x=[1]*100" "sum(x)" 10 loops, best of 3: 4.87 usec per loop python -mtimeit --setup="from operator import add; x=[1]*100" "reduce(add,x)" 10 loops, best of 3: 10.1 usec per loop # Adding floats python -mtimeit --setup="x=[1.0]*100" "sum(x)" 10 loops, best of 3: 5.14 usec per loop python -mtimeit --setup="from operator import add; x=[1.0]*100" "reduce(add,x)" 10 loops, best of 3: 10.1 usec per loop # Adding tuples python -mtimeit --setup="x=[(1,)]*100" "sum(x,())" 1 loops, best of 3: 61.6 usec per loop python -mtimeit --setup="from operator import add; x=[(1,)]*100" "reduce(add,x,())" 1 loops, best of 3: 66.3 usec per loop # Adding lists python -mtimeit --setup="x=[[1]]*100" "sum(x,[])" 1 loops, best of 3: 79.9 usec per loop python -mtimeit --setup="from operator import add; x=[[1]]*100" "reduce(add,x,[])" 1 loops, best of 3: 84.3 usec per loop George -- http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
On Sun, 04 May 2008 00:31:01 +0200, Thomas Dybdahl Ahle wrote: > On Sat, 2008-05-03 at 21:37 +, Ivan Illarionov wrote: >> On Sat, 03 May 2008 20:44:19 +0200, Szabolcs Horvát wrote: >> >> > Arnaud Delobelle wrote: >> >> >> >> sum() works for any sequence of objects with an __add__ method, not >> >> just floats! Your algorithm is specific to floats. >> > >> > This occurred to me also, but then I tried >> > >> > sum(['abc', 'efg'], '') >> >> Interesting, I always thought that sum is like shortcut of >> reduce(operator.add, ...), but I was mistaken. >> >> reduce() is more forgiving: >> reduce(operator.add, ['abc', 'efg'], '' ) # it works 'abcefg' > > Hm, it works for lists: > sum(([1], [2]), []) > [1, 2] > > However I find the seccond argument hack ugly. Does the sum way have any > performance advantages over the reduce way? Yes, sum() is faster: $ python -m timeit "" "sum([[1], [2], [3, 4]], [])" 10 loops, best of 3: 6.16 usec per loop $ python -m timeit "import operator" \ "reduce(operator.add, [[1], [2], [3, 4]])" 10 loops, best of 3: 11.9 usec per loop -- Ivan -- http://mail.python.org/mailman/listinfo/python-list
Re: dict invert - learning question
Assuming all the values are unique:
>>> a={1:'a', 2:'b', 3:'c'}
>>> dict(zip(a.keys(), a.values()))
{1: 'a', 2: 'b', 3: 'c'}
The problem is you obviously can't assume that in most cases.
Still, zip() is very useful function.
--
http://mail.python.org/mailman/listinfo/python-list
Re: dict invert - learning question
On 4 Maj, 01:27, [EMAIL PROTECTED] wrote:
> >>> a={1:'a', 2:'b', 3:'c'}
Oops, it should obviously be:
>>> dict(zip(a.values(), a.keys()))
{'a': 1, 'c': 3, 'b': 2}
--
http://mail.python.org/mailman/listinfo/python-list
generator functions in another language
I'm actually curious if there's a way to write a generator function (not a generator expression) in C, or what the simplest way to do it is... besides link the Python run-time. -- http://mail.python.org/mailman/listinfo/python-list
Re: PIL JPEG mis-step
On Sat, 03 May 2008 15:25:35 -0700, darkblueB wrote: > I got the Python Imaging Library from source, built and installed, on > Ubuntu 7.10, not realizing I could run a self-test first. libjpeg is on > the machine, but was not detected.. so no JPG encoder. I got the > dev-libjpg and rebuilt PIL. The self-test now shows JPG support. > > but running setup.py install again seems to succeed, but still no JPG > :-( > > fix ideas appreciated Make sure that libjpeg62-dev is installed 'sudo apt-get install libjpeg62-dev' Try run 'python setup.py build_ext -f' to force setup.py to rebuild everything with JPEG. And 'sudo python setup.py install' should install PIL with JPEG support. It should work on Ununtu 7.10 (works for me at least). However, if you want Tkinter support, you should edit setup.py and change TCL_ROOT variable: TCL_ROOT = '/usr/lib/tcl8.4', '/usr/include/tcl8.4' And make sure that you have tcl/tk libraries/headers: sudo apt-get install tcl8.4-dev tk8.4-dev -- Ivan -- http://mail.python.org/mailman/listinfo/python-list
Re: Light slices + COW
On May 3, 1:59 pm, [EMAIL PROTECTED] wrote:
> Sometimes different languages suggests me ways to cross-pollinate
> them.
>
> (Note: probably someone has already suggested (and even implemented)
> the following ideas, but I like to know why they aren't fit for
> Python).
>
> Python generators now allow me to package some code patterns common in
> my code, that other (older) languages didn't allow me to extract; for
> example given a sequence (array, list, etc) of items now and then I
> need to scan all the upper triangle of their cross matrix:
>
> def xpairs(seq):
>
> len_seq = len(seq)
>
> for i, e1 in enumerate(seq):
>
> for j in xrange(i+1, len_seq):
>
> yield e1, seq[j]
>
> Or adjacent ones (there are ways to generalize the following function,
> but this the most common situation for me):
>
> def xpairwise(iterable):
>
> return izip(iterable, islice(iterable, 1, None))
>
> That xpairs() generator is nice, but it's not the best possible code
> (but you may disagree with me, and you may think that code better than
> the successive D code that uses two slices). Inside it I can't use a
> list slicing because it copies subparts of the list, probably becoming
> too much slow (for example if len(seq) == 1000).
>
> Compared to Python, the D language is at lower level, so you may
> expect it to have a worse syntax (but the ShedSkin compiler has shown
> me once for all that you can have a language that is both high-level,
> with a short & sexy syntax, and very fast), but you can define a
> xpairs() iterable class in it too, and the core of that iterable class
> there's a method that may look like this:
>
> if (this.seq.length > 1)
>
> foreach (i, e1; this.seq[0 .. $-1])
>
> foreach (e2; this.seq[i+1 .. $]) {
>
> result = dg(e1, e2); if (result) break; // yield
>
> }
>
> Where the strange last line is the yield, and the $ inside [] is the
> length of the current array (a very clever idea that avoids the need
> for negative indexes).
>
> That D code is as fast or faster than the code you can back-translate
> from Python, this is possible because in D arrays slices are very
> light, they are a struct of (in the future they may
> change to a couple of pointers, to speed up the array slice scanning).
> So if you slice an array you don't copy its contents, just the start-
> end points of the slice. If you read/scan the slice that's all you
> have, while if you write on it, D uses a Copy-On-Write strategy, that
> is a just-in-time copy of the slice contents.
>
> In practice this speeds up lot of code that manages strings and arrays
> (like a XML parser, making it among the faster ones).
>
> Being the slice a struct, it's stack allocated, so the inner foreach
> doesn't even create a slice object in the heap each time the outer
> foreach loops.
>
> One problem with this strategy is that if you have a huge string, and
> you keep only a little slice of it, the D garbage collector will keep
> it all in memory. To avoid that problem you have to duplicate the
> slice manually:
>
> somestring[inf...sup].dup;
>
> I think Psyco in some situations is able to manage string slices
> avoiding the copy.
>
> I think Python 3.0 too may enjoy a similar strategy of light slices +
> COW for strings, lists and arrays (tuples and strings don't need the
> COW).
>
> Bye,
>
> bearophile
In my understanding, the operating system links files across multiple
sections on a disk, while keeping those details from client software.
Files:
AA C
While File A still reads as: AA, correctly.
Modifications to B as follow:
Files:
AA C
In the case of a large mutable string, modifications can cause linear-
time operations, even if only making a constant-time change.
String:
abcdefg
Modification:
aAbcdefg
causes the entire string to be recopied. Expensive at scale.
A string-on-disk structure could provide linking, such as a String
File Allocation Table.
abcdefg A
correctly reads as:
aAbcdefg
--
http://mail.python.org/mailman/listinfo/python-list
Re: PIL JPEG mis-step
On May 3, 4:52 pm, Ivan Illarionov <[EMAIL PROTECTED]> wrote: > Try run 'python setup.py build_ext -f' to force setup.py to rebuild > everything with JPEG. And 'sudo python setup.py install' should install > PIL with JPEG support. yes, that works (the self test still gives misleading results ?) but running some sample program works fine thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: dict invert - learning question
On May 3, 5:12 pm, dave <[EMAIL PROTECTED]> wrote:
> thanks Duncan and Arnaud.
>
> I'm learning Python from the "How to Think Like a Python Programmer"
> book by Allen Downey. My first try used the "inv[val] = [key]" and
> then the next problem was to incorporate the "D.setdefault(...)" method.
>
> Thank you for your help. I'm always amazed how kind people are in this group.
>
> On 2008-05-03 14:57:29 -0600, Arnaud Delobelle <[EMAIL PROTECTED]> said:
>
> > dave <[EMAIL PROTECTED]> writes:
>
> >> Hello,
>
> >> here is a piece of code I wrote to check the frequency of values and
> >> switch them around to keys in a new dictionary. Just to measure how
> >> many times a certain key occurs:
>
> >> def invert(d):
> >> inv = {}
> >> for key in d:
> >> val = d[key]
> >> if val not in inv:
> >> inv.setdefault(val, [key])
> > You can simply write:
> > inv[val] = [key]
> >> else:
> >> inv[val].append(key)
> >> return inv
>
> >> Using the methods above (I'm just a beginner) could I have written it
> >> more concisely? Any criticism/critique in the code would be greatly
> >> appreciated.
>
> > Apart from the unnecessary use of setdefault, it looks good to me.
>
> > * You could change if 'val not in inv:' to 'if val in inv:' (and swap
> > the if and else clauses of course) in order to have a positive
> > condition rather than a negative one
>
> > * If you want to use setdefault, you can replace the if .. else
> > construct by:
>
> > inv.setdefault(val, []).append(key)
>
> > * You can also iterate over keys and values using the items() or
> > iteritems() method of dictionaries:
>
> > def invert(d):
> > inv = {}
> > for key, val in d.iteritems():
> > inv.setdefault(val, []).append(key)
> > return inv
>
>
In Python 2.5 and later you may use the defaultdict class which is
faster and slightly more elegant in such cases:
from collections import defaultdict
def invert(d):
inv = defaultdict(list)
for key, val in d.iteritems():
inv[val].append(key)
return inv
George
--
http://mail.python.org/mailman/listinfo/python-list
Re: Feature suggestion: sum() ought to use a compensated summation algorithm
On Sat, 03 May 2008 17:43:57 -0700, George Sakkis wrote: > On May 3, 7:12 pm, Ivan Illarionov <[EMAIL PROTECTED]> wrote: >> On Sun, 04 May 2008 00:31:01 +0200, Thomas Dybdahl Ahle wrote: >> > On Sat, 2008-05-03 at 21:37 +, Ivan Illarionov wrote: >> >> On Sat, 03 May 2008 20:44:19 +0200, Szabolcs Horvát wrote: >> >> >> > Arnaud Delobelle wrote: >> >> >> >> sum() works for any sequence of objects with an __add__ method, >> >> >> not just floats! Your algorithm is specific to floats. >> >> >> > This occurred to me also, but then I tried >> >> >> > sum(['abc', 'efg'], '') >> >> >> Interesting, I always thought that sum is like shortcut of >> >> reduce(operator.add, ...), but I was mistaken. >> >> >> reduce() is more forgiving: >> >> reduce(operator.add, ['abc', 'efg'], '' ) # it works 'abcefg' >> > Hm, it works for lists: >> > sum(([1], [2]), []) >> > [1, 2] > > So it's not reduce() that is more forgiving, it's sum() that makes an > exception for strings only. > > >> > However I find the seccond argument hack ugly. Does the sum way have >> > any performance advantages over the reduce way? >> >> Yes, sum() is faster: >> >> $ python -m timeit "" "sum([[1], [2], [3, 4]], [])" 10 loops, best >> of 3: 6.16 usec per loop >> >> $ python -m timeit "import operator" \ "reduce(operator.add, [[1], [2], >> [3, 4]])" 10 loops, best of 3: 11.9 usec per loop > > Not really; you measure the import and the attribute access time in the > second case. sum() is 2x faster for adding numbers only: > > # Adding integers > python -mtimeit --setup="x=[1]*100" "sum(x)" 10 loops, best of 3: > 4.87 usec per loop python -mtimeit --setup="from operator import add; > x=[1]*100" "reduce(add,x)" > 10 loops, best of 3: 10.1 usec per loop > > # Adding floats > python -mtimeit --setup="x=[1.0]*100" "sum(x)" 10 loops, best of 3: > 5.14 usec per loop python -mtimeit --setup="from operator import add; > x=[1.0]*100" "reduce(add,x)" > 10 loops, best of 3: 10.1 usec per loop > > # Adding tuples > python -mtimeit --setup="x=[(1,)]*100" "sum(x,())" 1 loops, best of > 3: 61.6 usec per loop python -mtimeit --setup="from operator import add; > x=[(1,)]*100" "reduce(add,x,())" > 1 loops, best of 3: 66.3 usec per loop > > # Adding lists > python -mtimeit --setup="x=[[1]]*100" "sum(x,[])" 1 loops, best of > 3: 79.9 usec per loop python -mtimeit --setup="from operator import add; > x=[[1]]*100" "reduce(add,x,[])" > 1 loops, best of 3: 84.3 usec per loop > > George Thanks for correction. Forget about --setup. -- Ivan -- http://mail.python.org/mailman/listinfo/python-list
Re: RegEx for matching brackets
To check a complete python expression use:
def check_open_close(expr):
try:
eval(expr)
except SyntaxError:
return False
else:
return True
This also ignores brackets in quotes, and checks <= & >= operators are
syntatically correct etc...
But is may have side effects... ;-)
eg.
check_open_close('__import__("os").system("echo rm -rf /") # OK')
c.f http://en.wikipedia.org/wiki/Scope_creep
N
--
http://mail.python.org/mailman/listinfo/python-list
Help - IDLE Debugger Source Checkbox??
I have been learning the IDLE Debugger and was wondering if I was missing something with the "Source" checkbox setting in the debugger. As I single-step through my program, I have to uncheck and then recheck this box in order to see the current line in the file editing window highlighted. Is there anyway continually see the current line being executed in the file editor, without unchecking and rechecking the source setting in the debugger? Thanks for any insight on this issue. -- http://mail.python.org/mailman/listinfo/python-list
Re: is +=1 thread safe
AlFire <[EMAIL PROTECTED]> writes: >> The threading module already has a function to return the number of Thread >> objects currently alive. > > I have threads within threads - so it does not suit me :-(. How about using a scalar numpy array? They are mutable, so I assume that x += 1 should be atomic. 'as -- http://mail.python.org/mailman/listinfo/python-list
Re: Python documentation
[EMAIL PROTECTED] wrote: I have been learning python for some time using the dive into python book. I am interested to know if anyone can recommend a book which covers more advanced topics like threading and potentially GUI style coding If you don't at least browse "The Python Cookbook," you are missing a chance at something you'll find great. Note, I am strongly of the opinion that what makes a book great is anachronistic, so check it out before you buy. I love information-dense books like Knuth, Strunk & White, and the original C manual. I know people who hate these. I think of them as technical poetry, rather than prose. My guess is that Alex Martelli (editor for the most recent cookbook) is not discoursive enough for some; I appreciate that fact and understand others are in the "we read Knuth so you don't have to" camp. personally, I've loved most hours I've spent on Knuth -- I'd hate to deprve anyone of that experience. -Scott David Daniels [EMAIL PROTECTED] -- http://mail.python.org/mailman/listinfo/python-list
Script Optimization
Can anyone provide some advice/suggestions to make a script more precise/efficient/concise, etc.? The script verifies checksums, renames dirs/files, edits checksum filenames, and htm page filenames, all from mp3cd format ([0-9][0-9] [0-9]xxx.yyy) to something more usable: for ripping an mp3cd to disk (everything works, but I think it's a bit bulky). It's too long to post here (160 lines) so here's the link: http://uppit.com/d2/CKOYHE/af78a6bd3e21a19d5871abb9b879/utils.py (if that doesn't work: http://uppit.com/CKOYHE) Thanks in advance, lev -- http://mail.python.org/mailman/listinfo/python-list
Re: generator functions in another language
On Sat, 03 May 2008 16:39:43 -0700, castironpi wrote: > I'm actually curious if there's a way to write a generator function > (not a generator expression) in C, or what the simplest way to do it > is... besides link the Python run-time. The reference implementation of Python is written in C, so obviously there must be a way to write something like generators in C. Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Re: dict invert - learning question
On Sat, May 3, 2008 at 4:29 PM, dave <[EMAIL PROTECTED]> wrote:
> here is a piece of code I wrote to check the frequency of values and switch
> them around to keys in a new dictionary. Just to measure how many times a
> certain key occurs:
>
> def invert(d):
> inv = {}
> for key in d:
> val = d[key]
> if val not in inv:
> inv.setdefault(val, [key])
> else:
> inv[val].append(key)
> return inv
>
>
> Using the methods above (I'm just a beginner) could I have written it more
> concisely? Any criticism/critique in the code would be greatly appreciated.
If you're using python 2.5, the collections module has the defaultdict
type that is useful for things like this.
from collections import defaultdict
def invert2(d):
inv = defaultdict(list)
for key, value in d.items():
inv[value] += key
return dict(inv)
If you don't mind returning a defaultdict instead of a regular
dictionary, you could just 'return inv'. The defaultdict is useful
for building up the frequency counts in the first place too:
def wordcount(wordlist):
wc = defaultdict(int)
for word in wordlist:
wc[word] += 1
return dict(wc)
--
Jerry
--
http://mail.python.org/mailman/listinfo/python-list
Re: Light slices + COW
> That xpairs() generator is nice, but it's not the best possible code > (but you may disagree with me, and you may think that code better than > the successive D code that uses two slices). Inside it I can't use a > list slicing because it copies subparts of the list, probably becoming > too much slow (for example if len(seq) == 1000). What do you mean by best possible? Most efficient? Most readable? And why don't you use islice? eg: def xpairs(seq): len_seq = len(seq) for i, e1 in enumerate(seq): for e2 in islice(seq, i+1, None): yield e1, e2 Here's a version which makes more use of itertools. It should be more efficient, but it's ugly :-) (this is my first time using itertools). def xpairs(seq): def _subfunc(): for i in xrange(len(seq)): e1 = seq[i] yield izip(repeat(e1), islice(seq, i+1, None)) return chain(*_subfunc()) > That D code is as fast or faster than the code you can back-translate > from Python, this is possible because in D arrays slices are very > light, they are a struct of D compiles to efficient machine code so Python is at a disadvantage even if you use the same syntax (see my first example). You can make the Python version faster, but beware of premature optimization. > I think Python 3.0 too may enjoy a similar strategy of light slices + > COW for strings, lists and arrays (tuples and strings don't need the > COW). What I'dlike to see is a rope[1] module for Python. I'ts in C++'s STL library[2], but I haven't found a Python version yet. [1] http://en.wikipedia.org/wiki/Rope_(computer_science) [2] http://www.sgi.com/tech/stl/Rope.html With a Python rope library you could do things like this: a = '' b = rope(a) # Contains a reference to a c = b[0:10] # Get a rope object d = b[10:20] # Get another rope object e = b + b # Get another rope object print e # Get the string representation of all the re-assembled sub-sections # And so on. In the above code there was only 1 copy of the huge string in memory. The rope objects only contain a tree of sub-operations (slices, concatenations, references to original sequences, etc). This shouldn't be too hard to implement. Does anyone know of an already-existing 'rope' module? David. -- http://mail.python.org/mailman/listinfo/python-list
Re: is +=1 thread safe
Alexander Schmolck wrote: AlFire <[EMAIL PROTECTED]> writes: The threading module already has a function to return the number of Thread objects currently alive. I have threads within threads - so it does not suit me :-(. How about using a scalar numpy array? They are mutable, so I assume that x += 1 should be atomic. I ended up with following: counter=[] counter=0 counter.append(None)counter+=1 counter.pop() counter-=1 len(counter)counter A. -- http://mail.python.org/mailman/listinfo/python-list
Re: is +=1 thread safe
Alexander Schmolck wrote: AlFire <[EMAIL PROTECTED]> writes: The threading module already has a function to return the number of Thread objects currently alive. I have threads within threads - so it does not suit me :-(. How about using a scalar numpy array? They are mutable, so I assume that x += 1 should be atomic. No NO NO! The only way to increment a variable in memory is through a three step process: Load a register from a memory location Increment the register Store the value back into memory. Ages ago, there were architectures that would do an increment on a memory location in an atomic way, but with the current (RISC) architectures these are three separate operations. A good compiler may be able to eliminate some of these operations by keeping the variable in question in a register, but this just worsens the problem with respect to threads since each thread has it's own set of register values. Gary Herron 'as -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
saving a webpage's links to the hard disk
Is there a good place to look to see where I can find some code that will help me to save webpage's links to the local drive, after I have used urllib2 to retrieve the page? Many times I have to view these pages when I do not have access to the internet. -- http://mail.python.org/mailman/listinfo/python-list
Re: about bsddb module
On 5月3日, 下午7时17分, cocobear <[EMAIL PROTECTED]> wrote: > How to deal with multiple databases in an file. I want to get the > content of several databases. > > it's the code I wrote: > > [EMAIL PROTECTED] ~]$ python > Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11) > [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 > Type "help", "copyright", "credits" or "license" for more information.>>> > importbsddb > >>> import os > >>> import time > > >>> db_file = 'native.db' > >>> db =bsddb.db.DB() > >>> db.open(db_file,bsddb.db.DB_UNKNOWN,bsddb.db.DB_RDONLY) > >>> dbs = db.keys() > >>> db.open(db_file,dbs[0],bsddb.db.DB_UNKNOWN,bsddb.db.DB_RDONLY) > >>> db.keys() > > ['\x01\x00\x00\x00', '\x02\x00\x00\x00', '\x03\x00\x00\x00', > '\x04\x00\x00\x00', '\x05\x00\x00\x00'] > > >>> db.open(db_file,dbs[1],bsddb.db.DB_UNKNOWN,bsddb.db.DB_RDONLY) > > the program stop here. Anybody can help me? -- http://mail.python.org/mailman/listinfo/python-list
word shifts
Hello,
I made a function that takes a word list (one word per line, text file)
and searches for all the words in the list that are 'shifts' of
eachother. 'abc' shifted 1 is 'bcd'
Please take a look and tell me if this is a viable solution.
def shift(word, amt):
ans = ''
for letter in word:
ans = ans + chr((ord(letter) - ord('a') + amt) % 26 + ord('a'))
return ans
def fileshift(x):
fin = open(x)
d = {}
for line in fin:
d[line.strip()] = [1]
for i in range(1, 26):
ite = shift(line.strip(), i)
if ite in d:
print ite
Any tips/suggestions/critiques greatly appreciated.. I'm trying to
teach myself Python (and still beginning) and would love any helpful
info.
thanks!
--
http://mail.python.org/mailman/listinfo/python-list
Re: generator functions in another language
En Sun, 04 May 2008 01:08:34 -0300, Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> escribió: > On Sat, 03 May 2008 16:39:43 -0700, castironpi wrote: > >> I'm actually curious if there's a way to write a generator function >> (not a generator expression) in C, or what the simplest way to do it >> is... besides link the Python run-time. > > The reference implementation of Python is written in C, so obviously there > must be a way to write something like generators in C. Yes and no. Generators are tied to frames, and frames execute Python code, not C. There is no simple way to write generators in C, but there are some generator-like examples in the itertools module. See this thread http://groups.google.com/group/comp.lang.python/browse_thread/thread/5f42f72f2d0e88fc/ -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: saving a webpage's links to the hard disk
En Sun, 04 May 2008 01:33:45 -0300, Jetus <[EMAIL PROTECTED]> escribió: > Is there a good place to look to see where I can find some code that > will help me to save webpage's links to the local drive, after I have > used urllib2 to retrieve the page? > Many times I have to view these pages when I do not have access to the > internet. Don't reinvent the wheel and use wget http://en.wikipedia.org/wiki/Wget -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: Error handling in SAX
[EMAIL PROTECTED] wrote: > (this is a repost, for it's been a while since I posted this text via > Google Groups and it plain didn't appear on c.l.py - if it did appear > anyway, apols) It did, although some people have added google groups to their kill file. > So I set out to learn handling three-letter-acronym files in Python, > and SAX worked nicely until I encountered badly formed XMLs, like with > bad characters in it (well Unicode supposed to handle it all but > apparently doesn't), If it's not well-formed, it's not XML. XML parsers are required to reject non well-formed input. In case it actually is well-formed XML and the problem is somewhere in your code but you can't see it through the SAX haze, try lxml. It also allows you to pass the expected encoding to the parser to override broken document encodings. http://codespeak.net/lxml/ Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: word shifts
En Sun, 04 May 2008 02:17:07 -0300, dave <[EMAIL PROTECTED]> escribió:
> Hello,
>
> I made a function that takes a word list (one word per line, text file)
> and searches for all the words in the list that are 'shifts' of
> eachother. 'abc' shifted 1 is 'bcd'
>
> Please take a look and tell me if this is a viable solution.
>
> def shift(word, amt):
> ans = ''
> for letter in word:
> ans = ans + chr((ord(letter) - ord('a') + amt) % 26 + ord('a'))
> return ans
>
> def fileshift(x):
> fin = open(x)
> d = {}
> for line in fin:
> d[line.strip()] = [1]
> for i in range(1, 26):
> ite = shift(line.strip(), i)
> if ite in d:
> print ite
>
>
> Any tips/suggestions/critiques greatly appreciated.. I'm trying to
> teach myself Python (and still beginning) and would love any helpful
> info.
First, looking at the code, you're evaluating line.strip() a lot of times; I'd
avoid it.
Looks like you're using a dictionary just for the keys - the set type is more
adequate here (see http://docs.python.org/lib/types-set.html ). In any case,
I'd use a value like None instead of [1]
But I'd use a different algorithm. Instead of generating all posible shifts for
a given word, I'd substract the newly read word from each previous words (here,
"substract two words" means substract the character code for corresponding
positions, modulo 26). Shifted words, when substracted, have the same number on
all positions.
--
Gabriel Genellina
--
http://mail.python.org/mailman/listinfo/python-list
Re: is +=1 thread safe
On May 3, 7:44 pm, Gary Herron <[EMAIL PROTECTED]> wrote: > Alexander Schmolck wrote: > > AlFire <[EMAIL PROTECTED]> writes: > > >>> The threading module already has a function to return the number of Thread > >>> objects currently alive. > > >> I have threads within threads - so it does not suit me :-(. > > > How about using a scalar numpy array? They are mutable, so I assume that x > > += > > 1 should be atomic. > > No NO NO! The only way to increment a variable in memory is through a > three step process: Yes, this will be atomic. It's a pretty good idea, in fact. The underlying increment operation is protected by the GIL; it could be three, forty, or a hundred steps and it'd be atomic at the Python level. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: word shifts
On May 4, 2:04 am, "Gabriel Genellina" <[EMAIL PROTECTED]> wrote:
> En Sun, 04 May 2008 02:17:07 -0300, dave <[EMAIL PROTECTED]> escribió:
>
>
>
> > Hello,
>
> > I made a function that takes a word list (one word per line, text file)
> > and searches for all the words in the list that are 'shifts' of
> > eachother. 'abc' shifted 1 is 'bcd'
>
> > Please take a look and tell me if this is a viable solution.
>
> > def shift(word, amt):
> > ans = ''
> > for letter in word:
> > ans = ans + chr((ord(letter) - ord('a') + amt) % 26 + ord('a'))
> > return ans
>
> > def fileshift(x):
> > fin = open(x)
> > d = {}
> > for line in fin:
> > d[line.strip()] = [1]
> > for i in range(1, 26):
> > ite = shift(line.strip(), i)
> > if ite in d:
> > print ite
>
> > Any tips/suggestions/critiques greatly appreciated.. I'm trying to
> > teach myself Python (and still beginning) and would love any helpful
> > info.
>
> First, looking at the code, you're evaluating line.strip() a lot of times;
> I'd avoid it.
> Looks like you're using a dictionary just for the keys - the set type is more
> adequate here (seehttp://docs.python.org/lib/types-set.html). In any case,
> I'd use a value like None instead of [1]
> But I'd use a different algorithm. Instead of generating all posible shifts
> for a given word, I'd substract the newly read word from each previous words
> (here, "substract two words" means substract the character code for
> corresponding positions, modulo 26). Shifted words, when substracted, have
> the same number on all positions.
A faster algorithm is to create a 'key' for each word, defined as the
tuple of ord differences (modulo 26) of consecutive characters. E.g.
the key of 'acf' is (2,3); 'c' is 2 positions after 'a' and 'f' 3
positions after 'c'. Shifted words (and only these) have the same key.
Here's a straightforward implementation that generates all the lists
of equally-shifted words:
from collections import defaultdict
def iter_shifted(words):
key2shifted = defaultdict(list)
for word in words:
ords = map(ord,word)
key = tuple((ords[i]-ords[i-1]) % 26
for i in xrange(1,len(ords)))
key2shifted[key].append(word)
for shifted in key2shifted.itervalues():
if len(shifted) > 1:
yield shifted
if __name__ == '__main__':
words = 'abc bef jas cba cde zab azy hkl'.split()
for shifted in iter_shifted(words):
print shifted
# *** output ***
#['bef', 'hkl']
#['abc', 'cde', 'zab']
#['cba', 'azy']
--
http://mail.python.org/mailman/listinfo/python-list
