On Thu, Dec 1, 2016 at 3:26 AM, BartC <b...@freeuk.com> wrote: > On 30/11/2016 16:16, Heli wrote: >> >> Hi all, >> >> Writing my ASCII file once to either of pickle or npy or hdf data types >> and then working afterwards on the result binary file reduced the read time >> from 80(min) to 2 seconds. > > > 240,000% faster? Something doesn't sound quite right! How big is the file > now? (The one that had been 40GB of text and contained 100M lines or > datasets, although file caching is just about a possibility.)
Seems reasonable to me. Coming straight off the disk cache. ChrisA -- https://mail.python.org/mailman/listinfo/python-list