Re: Getting values out of a CSV
On Jul 14, 5:55 am, "Gabriel Genellina" <[EMAIL PROTECTED]> wrote: > > So, as always, one should measure in each specific case if optimization is > worth the pain [...]. > I hope I am somehow misreading the above sentence :-). IMO synonim language contructs should result in the same performance or at least have clear/ documented performance. I don't think we really want to see in code something like: if threshold: do_it_with_list_function else: do_it_with_list_comprehension bests, ./alex -- .w( the_mindstorm )p. > -- > Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
zipfile 2GB problems?
Hi fellas, I am experiencing problems reading a 2GB zipfile consisting of multiple zipped files. I found a thread http://mail.python.org/pipermail/python-dev/2005-April/053027.html that mentions a problem on the writing side, does such a problem exist on a reading side? I am using 2.4.1, perhaps there is a fix in a later version? -- http://mail.python.org/mailman/listinfo/python-list
Re: CSV without first line?
On Jul 15, 1:30 pm, "Sebastian Bassi" <[EMAIL PROTECTED]>
wrote:
> Hi,
>
> In my CSV file, the first line has the name of the variables. So the
> data I want to parse resides from line 2 up to the end. Here is what I
> do:
>
> import csv
> lines=csv.reader(open("MYFILE"))
> lines.next() #this is just to avoid the first line
> for line in lines:
> DATA PARSING
>
> This works fine. But I don't like to do "lines.next()" just to get rid
> of the first line. So I wonder if the reader function on the csv
> module has something that could let me parse the file from the second
> line (w/o doing that lines.next()).
>
Why do you wonder, instead of looking in the manual?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Marshalling big objects
En Sat, 14 Jul 2007 20:04:21 -0300, Orlando Döhring <[EMAIL PROTECTED]> escribió: > I want to marshal objects: > > - http://docs.python.org/lib/module-marshal.html > where I have problems with a bigger objects, e.g. Any specific reason you use this module? As a general purpose serializer, use pickle (or cPickle) instead. -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
On 7/13/07, Simon Hibbs <[EMAIL PROTECTED]> wrote: > place. At the end of it you'll have a good idea how OOP works, and how > Python works. Learning OOp this way is easy and painless, and what you ... But this tutorial states "I assume you know how object-oriented programming works" -- Sebastián Bassi (セバスティアン) Diplomado en Ciencia y Tecnología. GPG Fingerprint: 9470 0980 620D ABFC BE63 A4A4 A3DE C97D 8422 D43D -- http://mail.python.org/mailman/listinfo/python-list
Re: Circular import problem
En Sat, 14 Jul 2007 14:44:05 -0300, bvdp <[EMAIL PROTECTED]> escribió: >> > But, I still don't understand how python can access a function in a >> > file I have NOT included. In this case, to get things to work, I DO >> > NOT "import MMA.grooves" but later in the module I access a function >> > with "xx=MMA.grooves.somefunc()" and it finds the function, and works >> > just fine. It shouldn't work. > I've just used an empty __init__.py file. I will have to read up a bit > more on packages and see what advantage there is to import in that. __init__.py is executed inside the package's namespace - whatever you import/define there, is available later as packagename.zzz > Which is considered good style? Loading at the top or loading when > needed? I mean, importing at the top of the module is the recommended practice. See http://www.python.org/dev/peps/pep-0008/ I try only to import locally: a) when it is slow and not always needed, and b) when the imported module is unrelated to the main purpose (e.g. import traceback inside some exception handlers). -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
Dennis Lee Bieber <[EMAIL PROTECTED]> writes: > Though I should have added that, in Python, the toolset tends to > be... just an editor... Much more than that. The common toolset I see used is: * A customisable, flexible, programmer's text editor * The online documentation in a web browser * A Python interactive shell * An automated build tool like 'make' * A programmable shell for ad-hoc tasks * A multi-session terminal program to run all these simultaneously What I *don't* see is some single all-in-one tool specific to programming a particular language. Having learned one good instance of each of the above, it seems silly to need to learn all of them again simply because one has started using Python as the programming language. -- \ "Room service? Send up a larger room." -- Groucho Marx | `\ | _o__) | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: defaultdicts pickling
En Sat, 14 Jul 2007 13:03:01 -0300, Yoav Goldberg <[EMAIL PROTECTED]> escribió: > I need to have a dictionary of dictionaries of numbers, and I would like > the > dictionaries to be defaultdicts, because it makes the code much nicer > (I want to be able to do: d['foo']['bar']+=1 ). > > So naturally, I used: > > d = defaultdict(lambda :defaultdict(int)) > > It works great, but now I can not pickle it. > > I could ofcourse used a real function and not a lambda, but this would > make > things (a) somewhat slower and (b) a bit ugly. (a) AFAIK, lambdas are true functions, just without name, nor docstrings, nor decorator support, and they only allow expressions. But the execution time should be similar: from collections import defaultdict def test_lambda(): d = defaultdict(lambda: defaultdict(int)) for i in range(100): for j in range(100): d[i%7][j%7] += 1 def test_func(): def func(): return defaultdict(int) d = defaultdict(func) for i in range(100): for j in range(100): d[i%7][j%7] += 1 C:\TEMP>python -m timeit -s "from lf import test_lambda" "test_lambda()" 10 loops, best of 3: 80.3 msec per loop C:\TEMP>python -m timeit -s "from lf import test_func" "test_func()" 10 loops, best of 3: 79.6 msec per loop (the difference being less than the time variation I get, so it has no significance) (b) I don't consider a one-line function so much ugly, but ugliness is subjective, so I won't comment further. > Is there another way of achieving the same behaviour, that allow for > pickling? If you insist: Convert the defaultdict into a plain dict before pickling: plain_dict = dict(d) and again into a defaultdict when unpickling: d=defaultdict(...); d.update(unpickled_dict) -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
CSV without first line?
Hi,
In my CSV file, the first line has the name of the variables. So the
data I want to parse resides from line 2 up to the end. Here is what I
do:
import csv
lines=csv.reader(open("MYFILE"))
lines.next() #this is just to avoid the first line
for line in lines:
DATA PARSING
This works fine. But I don't like to do "lines.next()" just to get rid
of the first line. So I wonder if the reader function on the csv
module has something that could let me parse the file from the second
line (w/o doing that lines.next()).
--
Sebastián Bassi (セバスティアン)
Diplomado en Ciencia y Tecnología.
GPG Fingerprint: 9470 0980 620D ABFC BE63 A4A4 A3DE C97D 8422 D43D
--
http://mail.python.org/mailman/listinfo/python-list
Re: Fastest way to convert a byte of integer into a list
On 7 13 , 6 34 , Godzilla <[EMAIL PROTECTED]> wrote: > Hello, > > I'm trying to find a way to convert an integer (8-bits long for > starters) and converting them to a list, e.g.: > > num = 255 > numList = [1,1,1,1,1,1,1,1] > > with the first element of the list being the least significant, so > that i can keep appending to that list without having to worry about > the size of the integer. I need to do this because some of the > function call can return a 2 lots of 32-bit numbers. I have to find a > way to transport this in a list... or is there a better way? my clone *bin* function from python3000 def _bin(n, count=32): """returns the binary of integer n, using count number of digits""" return ''.join([str((n >> i) & 1) for i in range(count-1, -1, -1)]) -- http://mail.python.org/mailman/listinfo/python-list
Re: Installing mod_python on mac os 10.4.7
On Jul 15, 10:06 am, Graham Dumpleton <[EMAIL PROTECTED]> wrote: > On Jul 15, 2:47 am, 7stud <[EMAIL PROTECTED]> wrote: > > > > > Themod_pythonmanual says this under section 2.1 Prerequisites: > > > -- > > In order to compilemod_pythonyou will need to have the include files > > for both Apache and Python, as well as the Python library installed on > > your system. If you installed Python and Apache from source, then you > > already have everything needed. However, if you are using prepackaged > > software (e.g. Red Hat Linux RPM, Debian, or Solaris packages from > > sunsite, etc) then chances are, you have just the binaries and not the > > sources on your system. Often, the Apache and Python include files and > > libraries necessary to compilemod_pythonare part of separate > > ``development'' package. If you are not sure whether you have all the > > necessary files, either compile and install Python and Apache from > > source, or refer to the documentation for your system on how to get > > the development packages. > > - > > > I installed Apache from source using these instructions: > > >http://switch.richard5.net/isp-in-a-box-v2/installing-apache-on-mac-o... > > > but I used a package to install python 2.4. The package was from > > here: > > >http://www.pythonmac.org/packages/ > > > and it's a "Universal binary version of Python that runs natively on > > PPC and Intel systems." > > > But my mac came with Python 2.3.5 pre-installed, so I wonder if I > > already have the necessary "include files for both Apache and Python, > > as well as the Python library" already installed. > > Have you actually tried to install mod_python? That would be the > quickest way of finding out. > > Because you are using an alternate Apache than the OS supplied one, > you will need to use the --with-apxs option to configure when building > Python. Whoops, --with-apxs option is to configure for mod_python, not Python. > Unless you really need Python 2.4, it is easier to use the OS > supplied version of Python. If you must use an alternate version, use > the --with-python option to configure for mod_python to tell it which > version. Depending on where that Python version is installed, you may > also have to fiddle the Apache 'envvars' file as well to get it to > work. > > Graham -- http://mail.python.org/mailman/listinfo/python-list
Re: Fastest way to convert a byte of integer into a list
On Jul 15, 11:12 am, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > On Jul 14, 5:49?pm, Paul McGuire <[EMAIL PROTECTED]> wrote: > > > > > On Jul 13, 3:46 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > > > > On Jul 13, 5:17 am, Paul McGuire <[EMAIL PROTECTED]> wrote: > > > > > On Jul 12, 5:34 pm, Godzilla <[EMAIL PROTECTED]> wrote: > > > > > > Hello, > > > > > > I'm trying to find a way to convert an integer (8-bits long for > > > > > starters) and converting them to a list, e.g.: > > > > > > num = 255 > > > > > numList = [1,1,1,1,1,1,1,1] > > > > > > with the first element of the list being the least significant, so > > > > > that i can keep appending to that list without having to worry about > > > > > the size of the integer. I need to do this because some of the > > > > > function call can return a 2 lots of 32-bit numbers. I have to find a > > > > > way to transport this in a list... or is there a better way? > > > > > Standing on the shoulders of previous posters, I put this together. > > > > > -- Paul > > > > But aren't we moving backwards? The OP did ask for the fastest way. > > > > I put this together (from other posters and my own): > > > > import gmpy > > > import time > > > > y = 2**177149 - 1 > > > > # init list of tuples by byte > > > bytebits = lambda num : [num >> i & 1 for i in range(8)] > > > bytes = [ tuple(bytebits(i)) for i in range(256) ] > > > # use bytes lookup to get bits in a 32-bit integer > > > bits = lambda num : sum((bytes[num >> i & 255] for i in range(0,32,8)), > > > ()) > > > # use base-2 log to find how many bits in an integer of arbitrary > > > length > > > from math import log,ceil > > > log_of_2 = log(2) > > > numBits = lambda num : int(ceil(log(num)/log_of_2)) > > > # expand bits to integers of arbitrary length > > > arbBits = lambda num : sum((bytes[num >> i & 255] for i in > > > range(0,numBits(num),8)),()) > > > t0 = time.time() > > > L = arbBits(y) > > > t1 = time.time() > > > print 'Paul McGuire algorithm:',t1-t0 > > > > t0 = time.time() > > > L = [y >> i & 1 for i in range(177149)] > > > t1 = time.time() > > > print ' Matimus algorithm:',t1-t0 > > > > x = gmpy.mpz(2**177149 - 1) > > > t0 = time.time() > > > L = [gmpy.getbit(x,i) for i in range(177149)] > > > t1 = time.time() > > > print ' Mensanator algorithm:',t1-t0 > > > > ##Paul McGuire algorithm: 17.483676 > > > ## Matimus algorithm: 3.28100013733 > > > ## Mensanator algorithm: 0.125 > > > Oof! Pre-calculating those byte bitmasks doesn't help at all! It > > would seem it is faster to use a single list comp than to try to sum > > together the precalcuated sublists. > > > I *would* say though that it is somewhat cheating to call the other > > two algorithms with the hardcoded range length of 177149, when you > > know this is the right range because this is tailored to fit the input > > value 2**177149-1. This would be a horrible value to use if the input > > number were something small, like 5. I think numBits still helps here > > to handle integers of arbitrary length (and only adds a slight > > performance penalty since it is called only once). > > I agree. But I didn't want to compare your numBits > against gmpy's numdigits() which I would normally use. > But since the one algorithm required a hardcoded number, > I thought it best to hardcode all three. > > I originally coded this stupidly by converting > the number to a string and then converting to > a list of integers. But Matimus had already posted > his which looked a lot better than mine, so I didn't. > > But then I got to wondering if Matimus' solution > requires (B**2+B)/2 shift operations for B bits. > > My attempt to re-code his solution to only use B > shifts didn't work out and by then you had posted > yours. So I did the 3-way comparison using gmpy's > direct bit comparison. I was actually surprised at > the difference. > > I find gmpy's suite of bit manipulation routines > extremely valuable and use them all the time. > That's another reason for my post, to promote gmpy. > > It is also extremely important to keep coercion > out of loops. In other words, never use literals, > only pre-coerced constants. For example: > > import gmpy > def collatz(n): > ONE = gmpy.mpz(1) > TWO = gmpy.mpz(2) > TWE = gmpy.mpz(3) > while n != ONE: > if n % TWO == ONE: > n = TWE*n + ONE > else: > n = n/TWO > collatz(gmpy.mpz(2**177149-1)) > > > > > -- Paul Try the following function; it works for arbitrary positive integers, doesn't need a 3rd party module, doesn't need to worry whether all that ceil/log/log floating-point stuffing about could result in an off- by-one error in calculating the number of bits, works with Python 1.5.2, and is competitive with Matimus's method. def listbits(n): result = [] app = result.append while n: app(n & 1) n = n >> 1 return result Cheers, John -- http://mail.python.org/mailman/listinfo/python-list
Re: Circular import problem
bvdp wrote: > before I moved other > imports around I was able to do the following: > > 1. NOT include MMA.gooves, > 2. call the function MMA.grooves.somefunc() > > and have it work. The import doesn't necessarily have to be in the same module where the attribute is used. The first time *any* module does 'import MMA.grooves', a grooves attribute gets put into MMA, which can be used by any module that has a reference to MMA. It's probably not a good idea to rely on this, though, and better to put an explicit 'import MMA.grooves' into every module that uses it. Importing something more than once does no harm. -- Greg -- http://mail.python.org/mailman/listinfo/python-list
Re: Fastest way to convert a byte of integer into a list
On Jul 14, 5:49?pm, Paul McGuire <[EMAIL PROTECTED]> wrote: > On Jul 13, 3:46 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > > > > > > > On Jul 13, 5:17 am, Paul McGuire <[EMAIL PROTECTED]> wrote: > > > > On Jul 12, 5:34 pm, Godzilla <[EMAIL PROTECTED]> wrote: > > > > > Hello, > > > > > I'm trying to find a way to convert an integer (8-bits long for > > > > starters) and converting them to a list, e.g.: > > > > > num = 255 > > > > numList = [1,1,1,1,1,1,1,1] > > > > > with the first element of the list being the least significant, so > > > > that i can keep appending to that list without having to worry about > > > > the size of the integer. I need to do this because some of the > > > > function call can return a 2 lots of 32-bit numbers. I have to find a > > > > way to transport this in a list... or is there a better way? > > > > Standing on the shoulders of previous posters, I put this together. > > > > -- Paul > > > But aren't we moving backwards? The OP did ask for the fastest way. > > > I put this together (from other posters and my own): > > > import gmpy > > import time > > > y = 2**177149 - 1 > > > # init list of tuples by byte > > bytebits = lambda num : [num >> i & 1 for i in range(8)] > > bytes = [ tuple(bytebits(i)) for i in range(256) ] > > # use bytes lookup to get bits in a 32-bit integer > > bits = lambda num : sum((bytes[num >> i & 255] for i in range(0,32,8)), > > ()) > > # use base-2 log to find how many bits in an integer of arbitrary > > length > > from math import log,ceil > > log_of_2 = log(2) > > numBits = lambda num : int(ceil(log(num)/log_of_2)) > > # expand bits to integers of arbitrary length > > arbBits = lambda num : sum((bytes[num >> i & 255] for i in > > range(0,numBits(num),8)),()) > > t0 = time.time() > > L = arbBits(y) > > t1 = time.time() > > print 'Paul McGuire algorithm:',t1-t0 > > > t0 = time.time() > > L = [y >> i & 1 for i in range(177149)] > > t1 = time.time() > > print ' Matimus algorithm:',t1-t0 > > > x = gmpy.mpz(2**177149 - 1) > > t0 = time.time() > > L = [gmpy.getbit(x,i) for i in range(177149)] > > t1 = time.time() > > print ' Mensanator algorithm:',t1-t0 > > > ##Paul McGuire algorithm: 17.483676 > > ## Matimus algorithm: 3.28100013733 > > ## Mensanator algorithm: 0.125 > > Oof! Pre-calculating those byte bitmasks doesn't help at all! It > would seem it is faster to use a single list comp than to try to sum > together the precalcuated sublists. > > I *would* say though that it is somewhat cheating to call the other > two algorithms with the hardcoded range length of 177149, when you > know this is the right range because this is tailored to fit the input > value 2**177149-1. This would be a horrible value to use if the input > number were something small, like 5. I think numBits still helps here > to handle integers of arbitrary length (and only adds a slight > performance penalty since it is called only once). I agree. But I didn't want to compare your numBits against gmpy's numdigits() which I would normally use. But since the one algorithm required a hardcoded number, I thought it best to hardcode all three. I originally coded this stupidly by converting the number to a string and then converting to a list of integers. But Matimus had already posted his which looked a lot better than mine, so I didn't. But then I got to wondering if Matimus' solution requires (B**2+B)/2 shift operations for B bits. My attempt to re-code his solution to only use B shifts didn't work out and by then you had posted yours. So I did the 3-way comparison using gmpy's direct bit comparison. I was actually surprised at the difference. I find gmpy's suite of bit manipulation routines extremely valuable and use them all the time. That's another reason for my post, to promote gmpy. It is also extremely important to keep coercion out of loops. In other words, never use literals, only pre-coerced constants. For example: import gmpy def collatz(n): ONE = gmpy.mpz(1) TWO = gmpy.mpz(2) TWE = gmpy.mpz(3) while n != ONE: if n % TWO == ONE: n = TWE*n + ONE else: n = n/TWO collatz(gmpy.mpz(2**177149-1)) > > -- Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: MaildirMessage
Steve Holden <[EMAIL PROTECTED]> writes: > What do you actually think > > for m in msg: > print m > > should do? Why do you believe that what you think it should do would > be a natural choice? I think it odd for a Message to support the mapping protocol. However, since that's what is announced, then I expect it to actually support the mapping protocol, which in my expectation includes iterating over the keys. -- \ "One of the most important things you learn from the internet | `\ is that there is no 'them' out there. It's just an awful lot of | _o__) 'us'." -- Douglas Adams | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: Installing mod_python on mac os 10.4.7
On Jul 15, 2:47 am, 7stud <[EMAIL PROTECTED]> wrote: > Themod_pythonmanual says this under section 2.1 Prerequisites: > > -- > In order to compilemod_pythonyou will need to have the include files > for both Apache and Python, as well as the Python library installed on > your system. If you installed Python and Apache from source, then you > already have everything needed. However, if you are using prepackaged > software (e.g. Red Hat Linux RPM, Debian, or Solaris packages from > sunsite, etc) then chances are, you have just the binaries and not the > sources on your system. Often, the Apache and Python include files and > libraries necessary to compilemod_pythonare part of separate > ``development'' package. If you are not sure whether you have all the > necessary files, either compile and install Python and Apache from > source, or refer to the documentation for your system on how to get > the development packages. > - > > I installed Apache from source using these instructions: > > http://switch.richard5.net/isp-in-a-box-v2/installing-apache-on-mac-o... > > but I used a package to install python 2.4. The package was from > here: > > http://www.pythonmac.org/packages/ > > and it's a "Universal binary version of Python that runs natively on > PPC and Intel systems." > > But my mac came with Python 2.3.5 pre-installed, so I wonder if I > already have the necessary "include files for both Apache and Python, > as well as the Python library" already installed. Have you actually tried to install mod_python? That would be the quickest way of finding out. Because you are using an alternate Apache than the OS supplied one, you will need to use the --with-apxs option to configure when building Python. Unless you really need Python 2.4, it is easier to use the OS supplied version of Python. If you must use an alternate version, use the --with-python option to configure for mod_python to tell it which version. Depending on where that Python version is installed, you may also have to fiddle the Apache 'envvars' file as well to get it to work. Graham -- http://mail.python.org/mailman/listinfo/python-list
www.OutpatientSurgicare.com/video/
www.OutpatientSurgicare.com/video/ Outpatient Doctors Surgery Center is committed to offering the healthcare the community needs. We offer patients a meaningful alternative to traditional surgery. This state-of-the-art outpatient surgery center, located in the heart of Orange County, at 10900 Warner Avenue, Suite 101A, Fountain Valley, Ca 92708, offers the latest innovations in outpatient surgery and technology. Please Call For Our Special Cash Discount Toll Free: 1-877-500-2525 Please Visit Our Websites: We offer extreme cosmetic surgery makeover packages. http://www.SurgeonToTheStars.com http://www.1cosmeticsurgery.com Specializing in the cure of hyperhidrosis, sweaty palms, underarm and foot sweating. http://www.CuresweatyPalms.com http://www.ControlExcessiveSweating.com No. 1 Weight Loss Surgery Center http://www.ControlWeightLossNow.com http://www.FreeLapBandSeminar.com Hernia Treatment Center http://www.HerniaDoc.com Take care of your feet http://www.CureFootPain.com The Experts in CARPAL TUNNEL SYNDROME http://www.CureHandPain.com Accidental Urine Leaks ? End Urinary Incontinence http://www.WomanWellnessCenter.com Hemorrhoid Treatment Center http://www.hemorrhoidtreatmentcenter.com -- http://mail.python.org/mailman/listinfo/python-list
Marshalling big objects
Dear community,
I want to marshal objects:
- http://docs.python.org/lib/module-marshal.html
where I have problems with a bigger objects, e.g.
self.contiguousSurfaceResidues, see below.
I was writing the object to file:
fDump1 = open('dump_13PK_C.dat','wb')
marshal.dump(self.contiguousSurfaceResidues, fDump1)
fDump1.close()
loading it again:
fDump1 = open('dump_13PK_C.dat','rb')
self.contiguousSurfaceResidues = marshal.load(fDump1)
fDump1.close()
which yields nonsense as output, i.e. square-symbols etc. How to handle large
objects? Thanks.
Yours,
Orlando
--
self.contiguousSurfaceResidues =
[[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 1. 1. 1. 1. 0. 0. 0. 0. 0. 1. 1. 1.
1. 1. 0. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0.]
...
[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. [ 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 1. 0. 0. 1. 1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1.
0.]]
--
--
Pt! Schon vom neuen GMX MultiMessenger gehört?
Der kanns mit allen: http://www.gmx.net/de/go/multimessenger
--
http://mail.python.org/mailman/listinfo/python-list
Re: Fastest way to convert a byte of integer into a list
On Jul 13, 3:46 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > On Jul 13, 5:17 am, Paul McGuire <[EMAIL PROTECTED]> wrote: > > > > > > > On Jul 12, 5:34 pm, Godzilla <[EMAIL PROTECTED]> wrote: > > > > Hello, > > > > I'm trying to find a way to convert an integer (8-bits long for > > > starters) and converting them to a list, e.g.: > > > > num = 255 > > > numList = [1,1,1,1,1,1,1,1] > > > > with the first element of the list being the least significant, so > > > that i can keep appending to that list without having to worry about > > > the size of the integer. I need to do this because some of the > > > function call can return a 2 lots of 32-bit numbers. I have to find a > > > way to transport this in a list... or is there a better way? > > > Standing on the shoulders of previous posters, I put this together. > > > -- Paul > > But aren't we moving backwards? The OP did ask for the fastest way. > > I put this together (from other posters and my own): > > import gmpy > import time > > y = 2**177149 - 1 > > # init list of tuples by byte > bytebits = lambda num : [num >> i & 1 for i in range(8)] > bytes = [ tuple(bytebits(i)) for i in range(256) ] > # use bytes lookup to get bits in a 32-bit integer > bits = lambda num : sum((bytes[num >> i & 255] for i in range(0,32,8)), > ()) > # use base-2 log to find how many bits in an integer of arbitrary > length > from math import log,ceil > log_of_2 = log(2) > numBits = lambda num : int(ceil(log(num)/log_of_2)) > # expand bits to integers of arbitrary length > arbBits = lambda num : sum((bytes[num >> i & 255] for i in > range(0,numBits(num),8)),()) > t0 = time.time() > L = arbBits(y) > t1 = time.time() > print 'Paul McGuire algorithm:',t1-t0 > > t0 = time.time() > L = [y >> i & 1 for i in range(177149)] > t1 = time.time() > print ' Matimus algorithm:',t1-t0 > > x = gmpy.mpz(2**177149 - 1) > t0 = time.time() > L = [gmpy.getbit(x,i) for i in range(177149)] > t1 = time.time() > print ' Mensanator algorithm:',t1-t0 > > ##Paul McGuire algorithm: 17.483676 > ## Matimus algorithm: 3.28100013733 > ## Mensanator algorithm: 0.125- Hide quoted text - > > - Show quoted text - Oof! Pre-calculating those byte bitmasks doesn't help at all! It would seem it is faster to use a single list comp than to try to sum together the precalcuated sublists. I *would* say though that it is somewhat cheating to call the other two algorithms with the hardcoded range length of 177149, when you know this is the right range because this is tailored to fit the input value 2**177149-1. This would be a horrible value to use if the input number were something small, like 5. I think numBits still helps here to handle integers of arbitrary length (and only adds a slight performance penalty since it is called only once). -- Paul -- http://mail.python.org/mailman/listinfo/python-list
Creating Packages
Hello, All- I have written a class that allows me to create and manipulate data segments (the type one deals with when reading/writing text files); validating fields, serialization, etc. I would like to put this together as a module and release it to the community while I expand on the features. Is there a preferred way to do this? I have read the docutil information but recall hearing about Python Eggs being the new way to do such things. I have no intention of starting/continuing another debate (vim/emacs, perl/python, take your pick), but would like information/help as to my next step. I believe this package would be helpful to those of us dealing with text files. How do I go about making it available? Thanks, --greg -- http://mail.python.org/mailman/listinfo/python-list
Re: Screen Scraping Question
jeffbg123 wrote: > The numbers are always rendered the same. So I don't know if OCR is a > necessary step. > > Also, what if I just got the data from the packets? Any disadvantages > to that? Any good python packet capturing libraries? > > Thanks > Packet capture is probably a bad idea for two reasons. 1) It will give you a lot of packets you don't need and at a OSI layer so low that reassembly is painful. Try writing a HTTP or SOCKS proxy and get the Flash game to use that proxy. 2) Many Flash games would probably store (in the Flash file) or generate their scenarios instead of retrieving them over the Internet. -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3107 and stronger typing (note: probably a newbie question)
Hendrik van Rooyen wrote:
> "Lenard Lindstrom" <[EMAIL PROTECTED]> wrote:
>
>
>> Pascal has no break, continue or return. Eiffel doesn't even have a
>> goto. In such imperative languages boolean variables are used a lot.
>
> Thanks did not know this.
>
>> from StringIO import StringIO
>> lines = StringIO("one\ntwo\nthree\nfour\n")
>> line_number = 0
>> eof = False
>> found = False
>> while not (eof or found):
>> line = lines.readline()
>> eof = line == ""
>> found = line == "three\n"
>> if not found:
>> line_number += 1
>>
>>
>> if found:
>> print "Found at", line_number
>> else:
>> print "Not found"
>> # prints "Found at 2"
>>
>
> All right, so it is possible, with some legerdemain,
> forcing the main loop to stop when the condition
> is met.
> However, I find the above as clear as mud, compared to:
>
> line_number = 0
>
> for line in my_file:
> line_number += 1
> if "the Target" in line:
> break
>
> print line_number, line
>
> - Hendrik
>
I am not defending Pascal or similar languages. Pascal was an
educational language designed to teach structured programming by
removing temptation. The same example in Pascal would be less
convoluted, thought, since Pascal has an explicit end-of-file test
function (No, this is not a Python feature request.) Personally I don't
miss the continuous retesting of conditions made necessary in Pascal.
---
Lenard Lindstrom
<[EMAIL PROTECTED]>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
On Sat, 14 Jul 2007 11:49:48 -0600, darren kirby <[EMAIL PROTECTED]> wrote: >quoth the Wayne Brehaut: > >> (I started with Royal McBee LGP 30 machine language (hex input) in >> 1958, and their ACT IV assembler later! Then FORTRAN IV in 1965. By >> 1967 I too was using (Burroughs) Algol-60, and 10 years later upgraded >> to (DEC-10) Simula-67.) >> >> Going---going--- > >Mel? Is that you? > >http://www.pbm.com/~lindahl/mel.html > Ha-ha! Thanks for that! Although I'm not Mel, the first program I saw running on the LGP-30 was his Blackjack program! In 1958 I took a Numerical Methods course at the University of Saskatchewan, and we got to program Newton's forward difference method for the LGP-30. Our "computer centre tour" was to the attic of the Physics building, where their LGP-30 was networked to a similar one at the Univeristy of Toronto (the first educational computer network in Canada!), and the operator played a few hands of Blackjack with the operator there to illustrate how useful computers could be. A few years later, as a telecommunications officer in the RCAF, I helped design (but never got to teach :-( ) a course in LGP-30 architecture and programming using both ML and ACT IV AL, complete with paper tape input and Charactron Tube (http://en.wikipedia.org/wiki/Charactron) output--handy, since this display was also used in the SAGE system. We weren't encouraged to use card games as examples, so used navigational and tracking problems involving fairly simple trigonometry. wwwayne >-d -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
On Sat, 14 Jul 2007 19:18:05 +0530, "Rustom Mody" <[EMAIL PROTECTED]> wrote: >On 7/14/07, Alex Martelli <[EMAIL PROTECTED]> wrote: >> >> OOP can be abused (particularly with deep or intricate inheritance >> structures). But the base concept is simple and clear: you can bundle >> state and behavior into a stateful "black box" (of which you may make as >> many instances, with independent state but equal behavior, as you need). >> > >Many years ago (86??) Wegner wrote a paper in OOPSLA called Dimensions >of Object Orientation in which he called the 'base concept' of 'bundle >of state and behavior' as 'object based' programming and >'object-oriented' as object-based + inheritance. Not quite--according to him: object-based + classes => class-based class-based + class inheritance => object-oriented I.e., "object-oriented = objects + classes + inheritance". This was not the, by then, standard definition: to be OO would require all four of: 1. modularity (class-based? object-based?) 2. inheritance (sub-classing) 3. encapsulation (information hiding) 4. polymorphism ((sub-) class-specific response to a message, or processing of a method) Unfortunately, most of the "definitions" (usually just hand-waving, loosey-goosey descriptions) found on the web include none--or only one or two--of these fundamental requirements by name, and are so loose that almost any proramming paradigm or style would be OO. >What Alex is saying is (in effect) that object-based is simple and >clear (and useful) whereas the object-orientation is subject to abuse. But OO is also simple and clear (if clearly defined and explained and illustrated and implemented), and ANY programming style is subject to abuse. During the hey-day of Pascal as an introductory programming language (as often misused as more than that) I found many often spent much of their time defining the data types their program would use. >This anyway is my experience: C++ programmers are distinctly poorer >programmers than C programmers -- for some strange reason closeness to >the machine has a salutary effect whereas the encouragment of >uselessly over-engineered programs makes worse programmers. But this is a tautology: "over-engineered" programs are, by definition or terminology, not a good thing--independent of what PL or style they're finally implemented in (assuming that by "engineering" you mean "design" or similar). Many of my Pascal students over-engineered their solutions to simple problems too? >GUI is one of those cases wherein inheritance actually helps people >produce better code but this is something of an exception. This seems to imply that the list of applications you have in mind or have worked on includes fewer domains that might profit from full OO instead of just OB. My guess is that there are many application domains in which analysts and programmers often think in an "OO way", but implement in just an OB way because of the PL they or their employer requires or prefers: in some--perhaps many--of these cases they have to do "manually" what OO would have automated. There is a problem, though, of (especially university and college) education and training in OOP "talking about" how glorious OO is, and requiring students to use OO techniques whether they're most appropriate or not (the "classes early" pedagogical mindset). And this problem is compounded by teaching introductory programming using a language like Java that requires one to use an OO style for even trivial programs. And by using one of the many very similar introductory texbooks that talk a lot about OO before actually getting started on programming, so students don't realize how trivial a program is required to solve a trivial problem, and hence look for complexity everywhere--whether it exists or not--and spend a lot of time supposedly reducing the complexity of an already simple problem and its method of solution. But as I noted above, a similar problem occurred with the crop of students who first learned Pascal: they often spent much of their time defining the data types their program would use, just as OO (especially "classes early") graduates tend to waste time "over-subclassing" and developing libraries of little-used classes. The answer is always balance, and having an extensive enough toolkit that one is not forced or encouraged to apply a programming model that isn't appropriate and doesn't help in any way (including maintainability). And starting with a language that doesn't brainwash one into believing that the style it enforces or implies is always the best--and texbooks that teach proper choice of programming style instead of rigid adherence to one. wwwayne -- http://mail.python.org/mailman/listinfo/python-list
Re: getting text inside the HTML tag
On Sat, Jul 14, 2007 at 05:47:22PM +, Nikola Skoric wrote: > I'm using sgmllib.SGMLParser to parse HTML. I have successfuly > parsed start tags by implementing start_something method. But, now > I have to fetch the string inside the start tag and end tag too. I > have been reading through SGMLParser documentation, but just can't > figure that out... You need to define handle_data. You may also want to look at HTMLParser in addition to the alternatives previously mentioned. http://docs.python.org/lib/module-sgmllib.html -- [Will [EMAIL PROTECTED]|http://www.lfod.us/] -- http://mail.python.org/mailman/listinfo/python-list
Re: getting text inside the HTML tag
On Jul 14, 12:47 pm, Nikola Skoric <[EMAIL PROTECTED]> wrote: > I'm using sgmllib.SGMLParser to parse HTML. I have successfuly parsed start > tags by implementing start_something method. But, now I have to fetch the > string inside the start tag and end tag too. I have been reading through > SGMLParser documentation, but just can't figure that out... can somebody > help? :-) > > -- > "Now the storm has passed over me > I'm left to drift on a dead calm sea > And watch her forever through the cracks in the beams > Nailed across the doorways of the bedrooms of my dreams" Oi! Try Beautiful Soup instead. That seems to be the defacto HTML parser for Python: http://www.crummy.com/software/BeautifulSoup/ You might find the minidom or lxml modules to your liking as well. Mike -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
quoth the Wayne Brehaut: > (I started with Royal McBee LGP 30 machine language (hex input) in > 1958, and their ACT IV assembler later! Then FORTRAN IV in 1965. By > 1967 I too was using (Burroughs) Algol-60, and 10 years later upgraded > to (DEC-10) Simula-67.) > > Going---going--- Mel? Is that you? http://www.pbm.com/~lindahl/mel.html -d -- darren kirby :: Part of the problem since 1976 :: http://badcomputer.org "...the number of UNIX installations has grown to 10, with more expected..." - Dennis Ritchie and Ken Thompson, June 1972 -- http://mail.python.org/mailman/listinfo/python-list
getting text inside the HTML tag
I'm using sgmllib.SGMLParser to parse HTML. I have successfuly parsed start tags by implementing start_something method. But, now I have to fetch the string inside the start tag and end tag too. I have been reading through SGMLParser documentation, but just can't figure that out... can somebody help? :-) -- "Now the storm has passed over me I'm left to drift on a dead calm sea And watch her forever through the cracks in the beams Nailed across the doorways of the bedrooms of my dreams" -- http://mail.python.org/mailman/listinfo/python-list
Re: Circular import problem
> > But, I still don't understand how python can access a function in a > > file I have NOT included. In this case, to get things to work, I DO > > NOT "import MMA.grooves" but later in the module I access a function > > with "xx=MMA.grooves.somefunc()" and it finds the function, and works > > just fine. It shouldn't work. > > That depends a bit on what is "MMA" and what is "grooves". > MMA.grooves means "look for an attribute named grooves inside MMA". If MMA > is a module, and MMA.grooves is a class/function defined inside the > module, you don't need to import it before using it. > But if MMA is a package, and MMA.grooves is a module inside that package, > you need to import it first (unless MMA/__init__.py does that already) In this case (and I should have been more clear) MMA is a package and grooves is a module. It is working now, but before I moved other imports around I was able to do the following: 1. NOT include MMA.gooves, 2. call the function MMA.grooves.somefunc() and have it work. The point I was trying to make was that something in my imports was causing the namespace to contain MMA.grooves.* when it hadn't been imported in that module. I suspect that my reordering effected some initialization of modules, probably the creation of some classes. But that is a guess. I've just used an empty __init__.py file. I will have to read up a bit more on packages and see what advantage there is to import in that. > > I have tried to delay the import, and that does work. But, from a > > stylistic view I really to like to have all my imports at the top of > > the module. Maybe some old assembler/C habits on my part. > > Sure, it's considered good style. Which is considered good style? Loading at the top or loading when needed? -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
On Fri, 13 Jul 2007 20:37:04 -0400, Steve Holden <[EMAIL PROTECTED]> wrote: >Aahz wrote: >> In article <[EMAIL PROTECTED]>, >> Chris Carlen <[EMAIL PROTECTED]> wrote: >>>From what I've read of OOP, I don't get it. >> >> For that matter, even using OOP a bit with C++ and Perl, I didn't get it >> until I learned Python. >> >>> The problem for me is that I've programmed extensively in C and .asm on >>> PC DOS way back in 1988. >> >> Newbie. ;-) >> >> (I started with BASIC in 1976.) >> >Newbie ;-) > >(I started with Algol 60 in 1967). Newbie ;-) (I started with Royal McBee LGP 30 machine language (hex input) in 1958, and their ACT IV assembler later! Then FORTRAN IV in 1965. By 1967 I too was using (Burroughs) Algol-60, and 10 years later upgraded to (DEC-10) Simula-67.) Going---going--- >>> Form 2: Use Python and PySerial and TkInter or wxWidgets. >>> >>> Pro: Cross-platform goal will likely be achieved fully. Have a >>> programmer nearby with extensive experience who can help. >>> Con: Must learn new language and library. Must possibly learn a >>> completely new way of thinking (OOP) not just a new language syntax. >>> This might be difficult. >> >> My experience is that learning GUI programming is difficult. Moreover, >> GUI programming in C involves a lot of boilerplate that can be automated >> more easily with Python. So I think this will be a better solution. >> >I used to write in C for the SunView platform (back in the days when the >GUI was integrated into the kernel as the only way to get acceptable >speed on the display). From what I remember, "Hello World" took about 40 >lines. > >The immense (relatively speaking: this was 1985) size of the libraries >required was one of the primary justifications for implementing shared >libraries. > >> Note very very carefully that Python does not require an OOP style of >> programming, but it will almost certainly be the case that you just >> naturally start using OOP techniques as you learn Python. > >That's very true. I still use a lot of (perhaps too much) procedural >coding, but driving the object-oriented libraries is a great way for a >noob to get started in OOP. > >regards > Steve -- http://mail.python.org/mailman/listinfo/python-list
Installing mod_python on mac os 10.4.7
The mod_python manual says this under section 2.1 Prerequisites: -- In order to compile mod_python you will need to have the include files for both Apache and Python, as well as the Python library installed on your system. If you installed Python and Apache from source, then you already have everything needed. However, if you are using prepackaged software (e.g. Red Hat Linux RPM, Debian, or Solaris packages from sunsite, etc) then chances are, you have just the binaries and not the sources on your system. Often, the Apache and Python include files and libraries necessary to compile mod_python are part of separate ``development'' package. If you are not sure whether you have all the necessary files, either compile and install Python and Apache from source, or refer to the documentation for your system on how to get the development packages. - I installed Apache from source using these instructions: http://switch.richard5.net/isp-in-a-box-v2/installing-apache-on-mac-os-x/ but I used a package to install python 2.4. The package was from here: http://www.pythonmac.org/packages/ and it's a "Universal binary version of Python that runs natively on PPC and Intel systems." But my mac came with Python 2.3.5 pre-installed, so I wonder if I already have the necessary "include files for both Apache and Python, as well as the Python library" already installed. -- http://mail.python.org/mailman/listinfo/python-list
defaultdicts pickling
Hello, I need to have a dictionary of dictionaries of numbers, and I would like the dictionaries to be defaultdicts, because it makes the code much nicer (I want to be able to do: d['foo']['bar']+=1 ). So naturally, I used: d = defaultdict(lambda :defaultdict(int)) It works great, but now I can not pickle it. I could ofcourse used a real function and not a lambda, but this would make things (a) somewhat slower and (b) a bit ugly. Is there another way of achieving the same behaviour, that allow for pickling? Thanks, Yoav -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
The Tkinter tutorial refrrred to is at http://infohost.nmt.edu/tcc/help/pubs/tkinter// and it is a great starting point ... Ron Stephens On Jul 14, 3:01 am, "Hendrik van Rooyen" <[EMAIL PROTECTED]> wrote: > "Chris Carlen" wrote: > > > Form 2: Use Python and PySerial and TkInter or wxWidgets. > > > Pro: Cross-platform goal will likely be achieved fully. Have a > > programmer nearby with extensive experience who can help. > > Con: Must learn new language and library. Must possibly learn a > > completely new way of thinking (OOP) not just a new language syntax. > > This might be difficult. > > This is the way to go. - Trust me on this. > > When you describe your history, it is almost an exact parallel to mine. > In my case, I have been doing real low level stuff (mostly 8031 assembler) > since 1982 or so. And then I found python in a GSM module (Telit), and > I was intrigued. > > I really appreciate your comments on OO - it parallels a lot of what I feel > as there is a lot of apparent BS that does not seem to "do anything" at first > sight. > > However- for the GUI stuff, there is an easily understood relationship > between > the objects and what you see on the screen - so its a great way of getting > into OO - as far as people like you and me will go with it, which is not very > far, as we tend to think in machine instructions... > > And for what its worth - you can programme assembler-like python, and it also > works. > > The best thing to do is just to spend a few days playing with say Tkinter. > I use a reference from the web written by John W Shipman at New Mexico > Tech - it is succinct and clear, and deserves more widespread publicity. > > Google for it - I have lost the link, although I still have the pdf file. > > You will also find the interactive prompt that you get when you type > python at a command prompt invaluable - it lets you play with and debug > small code snippets so that you can learn as you go along - it really speeds > up the whole learning process, and makes it almost painless. > > All this talking is just wasting time - you could have had your first frame up > on the screen already, with a blank canvas, ready for drawing. It really goes > that quick, once you start. > > So the answer to the title question is: Yes - a low level programmer can learn > OOP, and its in fact easier than it looks, as almost all the heavy lifting has > been done for you by others. > > - Hendrik -- http://mail.python.org/mailman/listinfo/python-list
Re: pattern match !
[EMAIL PROTECTED] wrote: >> A slightly more generic match in case your package names turn out to be less >> consistent than given in the test cases: >> >> #!/usr/bin/python >> >> import re >> pattern = re.compile(r'(\w+?-(\d+[\.-])+\d+?)-\D+.*RPM') >> pkgnames = ["hpsmh-1.1.1.2-0-RHEL3-Linux.RPM", >> "hpsmh-1.1.1.2-RHEL3-Linux.RPM"] >> for pkg in pkgnames: >> matchObj = pattern.search(pkg) >> if matchObj: >> print matchObj.group(1) >> >> Still assumes it will end in RPM (all caps), but if you add the flag "re.I" >> to the re.compile() call, it will match case-insensitive. >> >> Hope that helps, >> >> -Jay > > How about if i had something like 1-3 words in the application name: > websphere-pk543-1.1.4.2-1-RHEL3-i386.rpm (in this case are 2 words)? Try this instead then: #!/usr/bin/python import re pattern = re.compile(r'((\w+?-)+?(\d+[\.-])+\d+?)-\D+.*RPM', re.I) pkgnames = ["hpsmh-1.1.1.2-0-RHEL3-Linux.RPM", "hpsmh-1.1.1.2-RHEL3-Linux.RPM", "websphere-pk543-1.1.4.2-1-RHEL3-i386.rpm"] for pkg in pkgnames: matchObj = pattern.search(pkg) if matchObj: print matchObj.group(1) -- http://mail.python.org/mailman/listinfo/python-list
Re: Access Denied while trying to delete file from python script
[EMAIL PROTECTED] wrote:
> On Jul 14, 3:39 am, "Viewer T." <[EMAIL PROTECTED]> wrote:
>> I am trying to write a script that deletes certain files based on
>> certain criteria.
>>
>> What I am trying to do is to automate the process of deleting certain
>> malware files that disguise themselves as system files and hidden
>> files. When I use os.remove() after importing the os module is raises
>> a Windows Error: Access denied probably because the files have
>> disguised themselves as system files. Is there anway to get adequate
>> privileges under windows to be able to delete system files from within
>> a python script. I know it is probably a windows security feature but
>> is there anyway to gain such privileges.
>
> You'll need to use the PyWin32 package (AKA win32all). There's a
> win32security module. These links might help:
>
> http://mail.python.org/pipermail/python-list/2006-February/367441.html
> http://www.thescripts.com/forum/thread28850.html
> http://aspn.activestate.com/ASPN/docs/ActivePython/2.5/pywin32/Windows_NT_Files_.2d.2d_Locking.html
>
> Sometimes you can beat windows over the head using its own delete
> commands:
>
> os.system('DEL /F /S /Q "%s"' % item)
> os.system(r'RD /S /Q "%s"' % item)
>
> I know there's a cookbook recipe on it, but I can't seem to find it
> right now. Hopefully this will help you some though.
I think what you want is this one:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/303343
I wrote/maintain an Win32 antivirus tool written in C and it uses the same
Win32 functions to unset any hidden/system attributes on files before
attempting to delete them. In the same vein, you'll find that some files are
locked and unable to be deleted, so you're probably want to look into the
following also:
* killing processes to remove currently running files
http://www.answermysearches.com/how-to-kill-a-process-by-name-in-python/57/
* scheduling files for removal on reboot using MoveFileEx
(you'd have to use win32 extensions to access the MoveFileEx function)
http://msdn2.microsoft.com/En-US/library/aa365240.aspx
Hope that helps,
-Jay
--
http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
On 7/14/07, Alex Martelli <[EMAIL PROTECTED]> wrote: > > OOP can be abused (particularly with deep or intricate inheritance > structures). But the base concept is simple and clear: you can bundle > state and behavior into a stateful "black box" (of which you may make as > many instances, with independent state but equal behavior, as you need). > Many years ago (86??) Wegner wrote a paper in OOPSLA called Dimensions of Object Orientation in which he called the 'base concept' of 'bundle of state and behavior' as 'object based' programming and 'object-oriented' as object-based + inheritance. What Alex is saying is (in effect) that object-based is simple and clear (and useful) whereas the object-orientation is subject to abuse. This anyway is my experience: C++ programmers are distinctly poorer programmers than C programmers -- for some strange reason closeness to the machine has a salutary effect whereas the encouragment of uselessly over-engineered programs makes worse programmers. GUI is one of those cases wherein inheritance actually helps people produce better code but this is something of an exception. And even here one of the most widely used popularisers of GUIs has been VB which was (at least initially) not object-oriented. VB shows that language orientation -- tailoring 'the language' of drag-n-drop to GUI-building and not just GUI-use -- wins over OOP. Ruby/Rails is another example of language-oriented programming though I feel it goes too far in (de)capitalizing, pluralizing, (de)hyphenizing etc towards 'readability'. [Sorry if this offends some people -- just my view!] And this makes me wonder: It seems that Tkinter, wxpython, pygtk etc are so much more popular among pythonistas than glade, dabo etc. Why is this? -- http://mail.python.org/mailman/listinfo/python-list
Re: Access Denied while trying to delete file from python script
On Jul 14, 3:39 am, "Viewer T." <[EMAIL PROTECTED]> wrote:
> I am trying to write a script that deletes certain files based on
> certain criteria.
>
> What I am trying to do is to automate the process of deleting certain
> malware files that disguise themselves as system files and hidden
> files. When I use os.remove() after importing the os module is raises
> a Windows Error: Access denied probably because the files have
> disguised themselves as system files. Is there anway to get adequate
> privileges under windows to be able to delete system files from within
> a python script. I know it is probably a windows security feature but
> is there anyway to gain such privileges.
You'll need to use the PyWin32 package (AKA win32all). There's a
win32security module. These links might help:
http://mail.python.org/pipermail/python-list/2006-February/367441.html
http://www.thescripts.com/forum/thread28850.html
http://aspn.activestate.com/ASPN/docs/ActivePython/2.5/pywin32/Windows_NT_Files_.2d.2d_Locking.html
Sometimes you can beat windows over the head using its own delete
commands:
os.system('DEL /F /S /Q "%s"' % item)
os.system(r'RD /S /Q "%s"' % item)
I know there's a cookbook recipe on it, but I can't seem to find it
right now. Hopefully this will help you some though.
Mike
--
http://mail.python.org/mailman/listinfo/python-list
Re: Client-side cookies on Python in Mac OSX
Adrian Petrescu <[EMAIL PROTECTED]> writes: > Oh, you're right! Silly me, I had always thought it was standard. > Thanks for pointing this out! I went and downloaded ClientCookie and > it works great on OS X. And since it is BSD-licensed, I can use it in > my app without any fear. Perfect. > > Thank you, Graham! The relationship is that cookielib is derived from ClientCookie: essentially the same code, cleaned up to take advantage of newer Python features, renamed to 'cookielib' and added to the stdlib. The other difference is in thread synchronisation: ClientCookie had some broken thread synchronisation at one point, which was subsequently removed (which doesn't imply ClientCookie is not useful in threaded code). Though an unfortunate series of events the released cookielib ended up with that broken thread synchronisation code. It seems at least some of those thread synchronisation bugs in cookielib are fixed in the lastest version of Python. http://python.org/sf/1484758 http://python.org/sf/1587139 John -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
[EMAIL PROTECTED] (Aahz) writes: [...] > Note very very carefully that Python does not require an OOP style of > programming, agree > but it will almost certainly be the case that you just > naturally start using OOP techniques as you learn Python. There's some truth to this. But stagnation is also very easy to achieve, without conscious effort to improve. Also, reading OOP books (and this list) is still beneficial, both before and after you've understood each concept: before because it helps to learn new concepts at a faster rate, and to learn concepts you'd otherwise miss; after because it helps "clean up" and extend your understanding and because it teaches you standard names for things, helping communication. John -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
[Chris Carlen] > From what I've read of OOP, I don't get it. I have also found some > articles profoundly critical of OOP. I tend to relate to these > articles. If you want to know the truth, and opt to neither trust a friend or colleague, nor spend the time to try it yourself, here's a third way: Compile Qt (a toolkit like wx or Tk) and watch the list of source file names scroll past. Beautiful! Perhaps there's some better way of doing GUIs, but watching that list of source files, one realises that that's an academic question: practically, OOP fits GUIs -- and much of the other code in Qt -- so well, and so much effort has been put into these GUI toolkit libraries, that one would be a fool not to use them right now. A somewhat separate issue: You'd also be a fool not to apply OOP to the GUI code *you* write *using* one of those OOP GUI toolkits. Though you won't learn that all at once or without conscious effort, that's not an obstacle with Python -- you can start small. Of course there's some level of experience / project size / project longevity / number of people involved below which dashing it off using what you know right now will be quicker, but the break-even point is not far off in your case, I think. [chris] > However, those articles were no more objective than the descriptions > of OOP I've read in making a case. Ie., what objective > data/studies/research indicates that a particular problem can be > solved more quickly by the programmer, or that the solution is more > efficient in execution time/memory usage when implemented via OOP > vs. procedural programming? [bruno] > None. Definitively. wrt/ developper time and memory, it's mostly a > matter of fit-your-brains. If it does, you'll find it easier, else [...] How do we have confidence that that's true without doing experiments? AFAIK, only a few such experiments have been done (not counting research that does not meet basic standards of competence or is not peer-reviewed). I think some programming techniques are simply better than others for certain tasks, even when including the variation in people's abilities (but excluding the cost of people learning those techniques, which can of course be significant). Of course, measurement is tricky because of differences between programmers, but it's not impossible. John -- http://mail.python.org/mailman/listinfo/python-list
Re: MaildirMessage
En Sat, 14 Jul 2007 06:26:43 -0300, Tzury <[EMAIL PROTECTED]> escribió: >> What do you actually think >> >> ... for m in msg: >> ... print m >> >> should do? Why do you believe that what you think it should do would be >> a natural choice? > > I needed to know how to extract particular parts of a message, such as > the message 'body', 'subject', 'author', etc. > I couldn't find it in the documentation this kind of information until > I ran the inspect.getmembers(msg) Well, you took the hard way... the easy way would be to read the documentation: http://docs.python.org/lib/module-email.message.html > and found out that the message body > stored in _payload. that is msg._payload is the content of the > message. You should perceive the initial _ in _payload as a yellow light - a warning. It's an implementation detail and one should avoid messing with it - best to stick to the documented API, get_payload() in this case. > Regarding the *bug* thing it is very simple. If you iterate an int you > get an exception , clear > enough. Yet here I got the following exception object has no attribute 'lower'> which suggest an attempt to iterate > ---. I do think that a message should support iteration for the very > fact that each can contain a different type and amount of headers, and > iteration in this case will make application's code clean and unified. I don't see what you mean. You can always enumerate the message headers using msg.keys(), along with has_key(), items(), etc. And what you want to unify the code with? Even if Message becomes iterable, I still don't see why should iteration over its *headers* be the preferred meaning. For a multipart/* MIME message, iterating over the parts looks much more interesting. -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: Right tool and method to strip off html files (python, sed, awk?)
[EMAIL PROTECTED] wrote: > 1- Find all html files in the folders (sub-folders ...) > 2- Do some file I/O and feed Sed or Python or what else with the file. > 3- Apply recursively some regular expression on the file to do the > things a want. (delete when it encounters certain tags, certain > attributes) > 4- Write the changed file, and go through all the files like that. Use the lxml.html.clean module, which is made exactly for that purpose. It's not released yet, but you can use it from the current html branch of lxml. There will soon be an official alpha of the 2.0 series, which will contain lxml.html: http://codespeak.net/svn/lxml/branch/html/ It looks like you're on Ubuntu, so compiling it from sources after an SVN checkout should be as simple as the usual setup.py dance. Please report back to the lxml mailing list if you find any problems or have any further ideas on how to make it even more versatile than it already is. For lxml is general, see: http://codespeak.net/lxml/ Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: help reading cookie values
[EMAIL PROTECTED] writes: > On Jul 13, 3:08 pm, Sean <[EMAIL PROTECTED]> wrote: >> I am trying to read a cookie I set but I am not sure if I really set >> it correctly or I am not reading it correctly. I was given the >> following instructions to set the cookie. It appears to be working >> because in Firefox browser I see the cookie listed for my domain [...snip cookie-sending and -receiving code...] > This site looks like it has some good information on Cookie handling > (see "cookielib and ClientCookie Example" section: > http://www.voidspace.org.uk/python/recipebook.shtml No, he doesn't want that -- that's for web clients, not web servers. Wild guess: are you sending a domain attribute in your cookie (sniff the HTTP traffic to find out, see below)? Try not doing that. The cookie-receiving code above looks correct (I don't know about the sending code, because I don't know what framework you're using, and don't want to read that framework's code). Remember that things like the cookie's domain and path have to match HTTP request's domain and path for cookies to be *returned* by the browser to the server. Simply having been *accepted* by the browser is not enough. The rules are are fairly complicated, so KISS. You can use a network sniffer like wireshark or tcpdump to record exactly what Set-Cookie: and Cookie: headers are being sent, and compare with the domain name, port, HTTP path and URL scheme (https/http) of the HTTP request that your browser is making. If you still can't figure it out, post those things here (change the domain name if you must, but keep the form of everything strictly unchanged). Or simplify your situation to make it easier to understand. Another source of help would be the mailing list for the web framework you're using. John -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
In article <[EMAIL PROTECTED]>, Chris Carlen <[EMAIL PROTECTED]> wrote: > Why would OOP be better? Different is not better. Popular is not > better. What the academics say is not better. Less lines of code might > be better, if the priority is ease of programming. Or, less machine > execution time or memory usage might be better, if that is the priority. Short answer: Increasing programmer productivity is better, and OO frequently accomplishes this. Longer answer: Consider OOP as one tool in the toolbox. It isn't "best" for every conceivable problem, but neither is procedural programming, functional programming, table driven state machines, or any other style of design and/or programming. Each works well in some situations and poorly in others. Having a large number of tools at your disposal, and knowing which ones to use, is a big plus. Let's talk about design versus implementation for a moment, since OO really applies to both, but in different ways. You mentioned state machines, and that is a good example of a design technique. You can look at a problem and convert it to a state machine (design), then implement the state machine in whatever language your computer understands. Over the years, finite state machines have proven to be very effective models because: 1) there are plenty of real world problems that map cleanly to a state machine 2) state machines are easy to implement reliably in most computer languages 3) if you have already implemented a state machine for problem A, then implementing it for problem B is pretty easy - the real work is translating problem B into a state machine OOD is similar. There are a large number of problems for which an object oriented design is a good fit. Once you have an OO design, you then need to implement it, and languages with good OO support make this a lot easier. >From what I have seen, the advantages of OO tend to increase with the size of the project. For example, polymorphism generally lets M clients work with N different kinds of objects by writing M+N chunks of code rather than M*N. When M or N is small, this difference in minor, but as M and N increase, it becomes significant. By combining state and function, objects provide a good means of encapsulating operations and keeping client code independent of lower level code. This is a very big win since it allows for the evolution of the lower level code without breaking all of the client code. As with polymorphism, the benefits of encapsulation tend to increase with the size of the project. Even before OO languages were popular, it was quite common to use some basic OO design in order to increase encapsulation. If you look at almost any GUI framework from the 80's or early 90's you'll find lots of C code with structs that the user is not supposed to mess with, and then functions that take pointers/handles/indexes to those "magic" structs as the first argument. This is really OO design implemented in a procedural language. In fact, GUI frameworks are an excellent example of a domain for which OO has established itself a very good way to model the problem. I could probably spend a lot more time on the merits of OO, but I think if you really want to understand its benefits you need to work with it in a domain for which OO is useful. It is possible that the specific projects you work on really wouldn't benefit much from OO, and that is why you haven't had the "a-ha!" moment. Forcing an OO model onto a problem just for the sake of OO will only result in frustration. Dave -- http://mail.python.org/mailman/listinfo/python-list
Re: MaildirMessage
> What do you actually think > > ... for m in msg: > ... print m > > should do? Why do you believe that what you think it should do would be > a natural choice? I needed to know how to extract particular parts of a message, such as the message 'body', 'subject', 'author', etc. I couldn't find it in the documentation this kind of information until I ran the inspect.getmembers(msg) and found out that the message body stored in _payload. that is msg._payload is the content of the message. Regarding the *bug* thing it is very simple. If you iterate an int you get an exception , clear enough. Yet here I got the following exception which suggest an attempt to iterate ---. I do think that a message should support iteration for the very fact that each can contain a different type and amount of headers, and iteration in this case will make application's code clean and unified. -- http://mail.python.org/mailman/listinfo/python-list
Re: how to install pygame package?
> There seems to be some problem. For the tar version, initial steps > execute OK, but after typing: > [EMAIL PROTECTED] pygame-1.7.1release]# ./configure > bash: ./configure: No such file or directory > > So i don't hav a configure file? What should i do now? Sorry, instead of ./configure do this: python setup.py install It's a python package after all :) But first I would recommend reading http://pygame.org/install.html -- http://mail.python.org/mailman/listinfo/python-list
Access Denied while trying to delete file from python script
I am trying to write a script that deletes certain files based on certain criteria. What I am trying to do is to automate the process of deleting certain malware files that disguise themselves as system files and hidden files. When I use os.remove() after importing the os module is raises a Windows Error: Access denied probably because the files have disguised themselves as system files. Is there anway to get adequate privileges under windows to be able to delete system files from within a python script. I know it is probably a windows security feature but is there anyway to gain such privileges. -- http://mail.python.org/mailman/listinfo/python-list
Re: interactive graphical script builder
nik wrote: > Hi, > > I am looking for an interactive graphical script builder for python. > Basically, something like the os X automator. I have made a group of > methods that some non-programmers need to combine into a script. I > don't need a python IDE necessarially, but more of a sequence builder. > I am imagining a tool that could show graphical representations of my > different commands and then string them together in a single "script." > I've done a lot of googling, and the closest things that I can find > are apple automator or labview, but before I go through the process of > making one of those interact with python I would like to know if > anybody already knows of something like them that is more tuned to > python. > > Thanks, > Nik > there's already someone who made a start to create a "free python labview", it's called Pyxel form Erik Lechak, an ogl-like interface. cheers, Stef Mientki -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
On Jul 14, 8:49 am, James Stroud <[EMAIL PROTECTED]> wrote: > > Any gui more complicated than a few entry fields and some checkbuttons > is going to lend itself to OOP--so if you want to do GUI, learn OOP. Yep, there is nothing to be added to that. Except maybe that if you don't care too much about the look&feel you may consider starting with Tkinter. Pros: 1. it is part of the standard library, and you already have it; 2. it is possibly the easiest/simplest GUI out there; 3. it runs pretty much everywhere with minimal fuss. Michele Simionato -- http://mail.python.org/mailman/listinfo/python-list
Direct Client - TAC Engineer/Support Engineer -Phoenix
Hello, We have requirement for TAC Engineer in Phoenix. If your skills and experience matches with the same, send me your resume asap with contact # and rate to '[EMAIL PROTECTED] Job Title= TAC Engineer Location = Phoenix Duration = 6months Must / Core Skills Linux System administrator Shell scripting Knowledge of Databases Administration The ideal candidate will have 5 or more years of support engineering experience, including 3 or more years supporting telecommunications firms and/or engineering experience in complex, large-scale IT and network environments. Technology experience should be current. Experience Requirements: Previous experience with professional services support organization (large Telco or Fortune 500 customers) or technical customer service environment, required. Programming or troubleshooting of complex software source code: Java, C, scripting languages Strong UNIX System administration and usage. Red Hat Linux and Sun Solaris. Network and complex system troubleshooting, including at least 2 years of large scale network troubleshooting. Prevailing network troubleshooting tools - sniffers, log analysis, scripting tools. 2+ years experience HTML, Java, JSP, Javascript, SOAP/Webservices, XML Use of SQL database to troubleshoot and isolate application level consistency issues. (Oracle PL/SQL) Source Control Management: Perforce, Subversion Bug tracking software: Bugzilla, DDTS Thanks and regards, Kan BTech Inc Recruiter 510-438-6834 [EMAIL PROTECTED] www.tbiinc.org -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3107 and stronger typing (note: probably a newbie question)
"Lenard Lindstrom" <[EMAIL PROTECTED]> wrote:
> Pascal has no break, continue or return. Eiffel doesn't even have a
> goto. In such imperative languages boolean variables are used a lot.
Thanks did not know this.
>
> from StringIO import StringIO
> lines = StringIO("one\ntwo\nthree\nfour\n")
> line_number = 0
> eof = False
> found = False
> while not (eof or found):
> line = lines.readline()
> eof = line == ""
> found = line == "three\n"
> if not found:
> line_number += 1
>
>
> if found:
> print "Found at", line_number
> else:
> print "Not found"
> # prints "Found at 2"
>
All right, so it is possible, with some legerdemain,
forcing the main loop to stop when the condition
is met.
However, I find the above as clear as mud, compared to:
line_number = 0
for line in my_file:
line_number += 1
if "the Target" in line:
break
print line_number, line
- Hendrik
--
http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
"Aahz" <[EMAIL PROTECTED]> wrote: > Newbie. ;-) > > (I started with BASIC in 1976.) > *grinz @ Newbie* I was writing COBOL and NEAT/3 in 1968... - Hendrik -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
"Chris Carlen" wrote: > Form 2: Use Python and PySerial and TkInter or wxWidgets. > > Pro: Cross-platform goal will likely be achieved fully. Have a > programmer nearby with extensive experience who can help. > Con: Must learn new language and library. Must possibly learn a > completely new way of thinking (OOP) not just a new language syntax. > This might be difficult. > This is the way to go. - Trust me on this. When you describe your history, it is almost an exact parallel to mine. In my case, I have been doing real low level stuff (mostly 8031 assembler) since 1982 or so. And then I found python in a GSM module (Telit), and I was intrigued. I really appreciate your comments on OO - it parallels a lot of what I feel as there is a lot of apparent BS that does not seem to "do anything" at first sight. However- for the GUI stuff, there is an easily understood relationship between the objects and what you see on the screen - so its a great way of getting into OO - as far as people like you and me will go with it, which is not very far, as we tend to think in machine instructions... And for what its worth - you can programme assembler-like python, and it also works. The best thing to do is just to spend a few days playing with say Tkinter. I use a reference from the web written by John W Shipman at New Mexico Tech - it is succinct and clear, and deserves more widespread publicity. Google for it - I have lost the link, although I still have the pdf file. You will also find the interactive prompt that you get when you type python at a command prompt invaluable - it lets you play with and debug small code snippets so that you can learn as you go along - it really speeds up the whole learning process, and makes it almost painless. All this talking is just wasting time - you could have had your first frame up on the screen already, with a blank canvas, ready for drawing. It really goes that quick, once you start. So the answer to the title question is: Yes - a low level programmer can learn OOP, and its in fact easier than it looks, as almost all the heavy lifting has been done for you by others. - Hendrik -- http://mail.python.org/mailman/listinfo/python-list
Re: Can a low-level programmer learn OOP?
Chris Carlen wrote: > Hi: > > From what I've read of OOP, I don't get it. I have also found some > articles profoundly critical of OOP. I've also found articles critical of Darwinism--but we can chalk that up to religious zealotry can't we? Any gui more complicated than a few entry fields and some checkbuttons is going to lend itself to OOP--so if you want to do GUI, learn OOP. The time you spend learning OOP will be about 1/10th the time required to debug a modestly complicated gui. This is especially true of guis that require real-time feedback behavior. If you just want to enter some values and set some flags and then hit "go", you could always program the GUI in HTML and have a cgi script process the result. This has a lot of benefits that are frequently overlooked but tend to be less fun than using a bona-fide toolkit like WX or QT. James -- http://mail.python.org/mailman/listinfo/python-list
Re: Can I change one line in a file without rewriting the whole thing?
J. J. Ramsey wrote: > if I can avoid doing what amounts to reading the > whole file into memory, changing the copy in memory, and writing it > all out again. Except in very special circumstances, not really. If you do anything that makes a line longer or shorter, everything after that line in the file needs to be re-written, one way or another. If you're doing this sort of thing a lot, and need it to be faster than reading and rewriting the file, you may need to look into using a more sophisticated format on disk than a plain file. -- Greg -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3107 and stronger typing (note: probably a newbie question)
Ben Finney <[EMAIL PROTECTED]> writes: > This seems to make the dangerous assumption that the programmer has > the correct program in mind, and needs only to transfer it correctly > to the computer. Well, I hope the programmer can at least state some clear specficiations that a correct program should implement, e.g. it should not freeze or crash. > I would warrant that anyone who understands exactly how a program > should work before writing it, and makes no design mistakes before > coming up with a program that works, is not designing a program of any > interest. I'm not sure what the relevance of this is. Programmers generally have an idea of what it is that they're trying to write. They then have to make a bunch of design and implementation decisions as they go along. Soemtimes those decisions are subtly incorrect, and the more errors the computer can identify automatically, the fewer errors are likely to remain in the program. > Certainly. Which is why the trend continues toward developing programs > such that mistakes of all kinds cause early, obvious failure -- where > they have a much better chance of being caught and fixed *before* > innocent hands get ahold of them. Here is what Edward Lee wrote about developing a multi-threaded Java app (http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf): A part of the Ptolemy Project experiment was to see whether effective software engineering practices could be developed for an academic research setting. We developed a process that included a code maturity rating system (with four levels, red, yellow, green, and blue), design reviews, code reviews, nightly builds, regression tests, and automated code coverage metrics [43]. The portion of the kernel that ensured a consistent view of the program structure was written in early 2000, design reviewed to yellow, and code reviewed to green. The reviewers included concurrency experts, not just inexperienced graduate students (Christopher Hylands (now Brooks), Bart Kienhuis, John Reekie, and myself were all reviewers). We wrote regression tests that achieved 100 percent code coverage. The nightly build and regression tests ran on a two processor SMP machine, which exhibited different thread behavior than the development machines, which all had a single processor. The Ptolemy II system itself began to be widely used, and every use of the system exercised this code. No problems were observed until the code deadlocked on April 26, 2004, four years later. It is certainly true that our relatively rigorous software engineering practice identified and fixed many concurrency bugs. But the fact that a problem as serious as a deadlock that locked up the system could go undetected for four years despite this practice is alarming. How many more such problems remain? How long do we need test before we can be sure to have discovered all such problems? Regrettably, I have to conclude that testing may never reveal all the problems in nontrivial Multithreaded code. Now consider that the code you test today on a 2-core processor may be running on a 100-core processor in 4 years, making any concurrency errors that much worse. You can't test on the 100-core cpu today because it doesn't exist yet, but your code still has to run on it correctly once it's out there. Heck, if you're writing in CPython (no parallelism at all, due to the GIL) you might be in for a surprise as soon as real parallelism arrives on even 2 cores. It gets worse, maybe you're writing operating system or language runtime code (e.g. parallel garbage collector for Java). That means there will be hostile code on the same machine, written by diabolical maniacs who know every detail of your implementation, and who deliberately do stuff (e.g. write code to hit specific cache lines or cause page faults at fiendishly precisely chosen moments) specifically to cause weird timing conditions and trigger bugs. Testing is not enough to ensure that such attacks won't succeed. I remember an old programming book (might even have been Knuth vol. 3) recommending using the Shell sorting algorithm (a very simple exchange sort that runs in about O(n**1.4) time) instead of an O(n log n) algorithm because Shell's algorithm (being simpler) was less likely to end up with implementation errors. Today I think an experienced programmer needing to code a sort routine wouldn't hesitate to code a good algorithm, because our languages and coding/testing methods are better than they were in the Fortran and assembly language era when that book was written. But there are still more complex problems (like shared memory multiprocessing) that we're advised to stay away from because they're so fraught with hazards. We instead use approaches that are simpler but take perforamnce penalties. (As a non-concurrency example, we might implement geometric search through som
