@Steven: Thank you...My input data is basically AUGC and newlines... I would like to know about bytearray technique. Please suggest me some links or reference.. I will go through the profiler and check whether the code maintains linearity with the input files.
> > It's probably worth putting some more timing statements into your code > > to see where the time is going because it's not the reading from the > > disk that's the problem. > > The first thing I would do is run the code on three smaller sample > files: > > 50MB > 100MB > 200MB > > The time taken should approximately double as you double the size of the > file: say it takes 2 hours to process the 50MB file, 4 hours for the > 100MB file and 8 hours for the 200 MB file, that's linear performance > and isn't too bad. > > But if performance isn't linear, say 2 hours, 4 hours, 16 hours, then > you're in trouble and you *desperately* need to reconsider the algorithm > being used. Either that, or just accept that this is an inherently slow > calculation and it will take a week or two. > > Amal, another thing you should try is use the Python profiler on your > code (again, on a smaller sample file). The profiler will show you where > the time is being spent. > > Unfortunately the profiler may slow your code down, so it is important > to use it on manageable sized data. The profiler is explained here: > > http://docs.python.org/3/library/profile.html > > If you need any help, don't hesitate to ask. > > > > >trying to optimize my code to get the outputs in less time and memory > > >efficiently. > > > > Memory efficiency is easy, do it line by line off the disk. > > This assumes that you can process one line at a time, sequentially. I > expect that is not the case. > > > -- > Steven > _______________________________________________ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > -- *AMAL THOMAS*
_______________________________________________ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor