Thanks Dan! I've never used CPython or PyPy. Will try them later. I think the main difference between your create_file_numbers_file_like() and the fastwrite5.py I sent earlier is that I've used cStringIO instead of StringIO. It took 12s less using cStringIO.
My numbers are much greater, but I've used Python 2.7.5 instead: C:\src\Python>python create_file_numbers.py time taken to write a file of size 52428800 is 39.1199457743 seconds time taken to write a file of size 52428800 is 14.8704800436 seconds time taken to write a file of size 52428800 is 23.0011990985 seconds I've downloaded bufsock.py and python2x3.py. The later one was hard to remove the source code from the web page. Can I use them on my projects? I'm not used to the UCI license[1]. What's the difference to the GPL? [1] http://stromberg.dnsalias.org/~dstromberg/UCI-license.html ________________________________ > Date: Sat, 18 May 2013 12:38:30 -0700 > Subject: Re: How to write fast into a file in python? > From: drsali...@gmail.com > To: lokeshkopp...@gmail.com > CC: python-list@python.org > > > With CPython 2.7.3: > ./t > time taken to write a file of size 52428800 is 15.86 seconds > > time taken to write a file of size 52428800 is 7.91 seconds > > time taken to write a file of size 52428800 is 9.64 seconds > > > With pypy-1.9: > ./t > time taken to write a file of size 52428800 is 3.708232 seconds > > time taken to write a file of size 52428800 is 4.868304 seconds > > time taken to write a file of size 52428800 is 1.93612 seconds > > Here's the code: > #!/usr/local/pypy-1.9/bin/pypy > #!/usr/bin/python > > import sys > import time > import StringIO > > sys.path.insert(0, '/usr/local/lib') > import bufsock > > def create_file_numbers_old(filename, size): > start = time.clock() > > value = 0 > with open(filename, "w") as f: > while f.tell() < size: > f.write("{0}\n".format(value)) > value += 1 > > end = time.clock() > > print "time taken to write a file of size", size, " is ", (end > -start), "seconds \n" > > def create_file_numbers_bufsock(filename, intended_size): > start = time.clock() > > value = 0 > with open(filename, "w") as f: > bs = bufsock.bufsock(f) > actual_size = 0 > while actual_size < intended_size: > string = "{0}\n".format(value) > actual_size += len(string) + 1 > bs.write(string) > value += 1 > bs.flush() > > end = time.clock() > > print "time taken to write a file of size", intended_size, " is ", > (end -start), "seconds \n" > > > def create_file_numbers_file_like(filename, intended_size): > start = time.clock() > > value = 0 > with open(filename, "w") as f: > file_like = StringIO.StringIO() > actual_size = 0 > while actual_size < intended_size: > string = "{0}\n".format(value) > actual_size += len(string) + 1 > file_like.write(string) > value += 1 > file_like.seek(0) > f.write(file_like.read()) > > end = time.clock() > > print "time taken to write a file of size", intended_size, " is ", > (end -start), "seconds \n" > > create_file_numbers_old('output.txt', 50 * 2**20) > create_file_numbers_bufsock('output2.txt', 50 * 2**20) > create_file_numbers_file_like('output3.txt', 50 * 2**20) > > > > > On Thu, May 16, 2013 at 9:35 PM, > <lokeshkopp...@gmail.com<mailto:lokeshkopp...@gmail.com>> wrote: > On Friday, May 17, 2013 8:50:26 AM UTC+5:30, > lokesh...@gmail.com<mailto:lokesh...@gmail.com> wrote: > > I need to write numbers into a file upto 50mb and it should be fast > > > > can any one help me how to do that? > > > > i had written the following code.. > > > > > ----------------------------------------------------------------------------------------------------------- > > > > > def create_file_numbers_old(filename, size): > > > > start = time.clock() > > > > > > > > value = 0 > > > > with open(filename, "w") as f: > > > > while f.tell()< size: > > > > f.write("{0}\n".format(value)) > > > > value += 1 > > > > > > > > end = time.clock() > > > > > > > > print "time taken to write a file of size", size, " is ", (end > -start), "seconds \n" > > > > > ------------------------------------------------------------------------------------------------------------------ > > > > > it takes about 20sec i need 5 to 10 times less than that. > size = 50mb > -- > http://mail.python.org/mailman/listinfo/python-list > > > -- http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list