On 22/08/12 22:23, Pete O'Connell wrote:
What makes you say it is "terribly slow"? Perhaps it is as fast as it
could be under the circumstances. (Maybe it takes a long time because
you have a lot of data, not because it is slow.)
OK maybe I am wrong about it being slow (I thought for loops were
slower than lis comprehensions).
For loops are slightly slower than list comprehensions but depending on
what is happening inside the loop/comprehension the difference may not
be significant.
> But I do know I need it to be as fast
> as possible if I need to run it on a thousand files each with hundreds
> of thousands of lines
If you need it as "fast as possible" you need to run it on a
supercomputer with a program in assembler after many hours of fine
tuning. But that will cost a lot of money and you probably don't really
need it.
For example how often do you need to run this process? Is it a one off -
set it running over lunch and it will likely be done by the time you get
back. Is it an hourly event? Set it running in the background while you
carry on processing the results of the previous run.
Steven's point is that usually absolute speed is not as important as
people think. Modern computers are absurdly fast. When I started using
Linux it took about 2 hours to build the kernel. Now I can build it in
about 2 minutes! Processing a thousand files is largely limited by the
speed of your hard drive rather than the speed of the processing on the CPU.
HTH
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
_______________________________________________
Tutor maillist - [email protected]
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
