On 04/11/13 16:34, Amal Thomas wrote:
@Joel: The code runs for weeks..input file which I have to process in
very huge(in 50 gbs). So its not a matter of hours.its matter of days
and weeks..
OK, but that's not down to reading the file from disk.
Reading a 50G file will only take a few minutes if you have enough RAM,
which seems to be the case. If it's taking days/weeks you must be doing
some incredibly time consuming processing.
It's probably worth putting some more timing statements into your code
to see where the time is going because it's not the reading from the
disk that's the problem.
trying to optimize my code to get the outputs in less time and memory
efficiently.
Memory efficiency is easy, do it line by line off the disk.
time efficiency is most likely down to your processing algorithm
if we are talking about days.
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.flickr.com/photos/alangauldphotos
_______________________________________________
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
