Re: write a 20GB file

2010-05-15 Thread Patrick Maupin
On May 15, 7:09 pm, Dave Angel wrote: > Nathan Rice wrote: > > This is precisely the situation mmap was made for :)  It has almost the same > > methods as a file so it should be an easy replacement. > > > > > Only on a 64bit system, and I'm not sure it's even possible there in > every case.  On a

Re: write a 20GB file

2010-05-15 Thread Patrick Maupin
On May 15, 7:09 pm, Dave Angel wrote: > Nathan Rice wrote: > > This is precisely the situation mmap was made for :)  It has almost the same > > methods as a file so it should be an easy replacement. > > > > > Only on a 64bit system, and I'm not sure it's even possible there in > every case.  On a

Re: write a 20GB file

2010-05-15 Thread Dave Angel
Nathan Rice wrote: This is precisely the situation mmap was made for :) It has almost the same methods as a file so it should be an easy replacement. Only on a 64bit system, and I'm not sure it's even possible there in every case. On a 32bit system, it would be impossible to mmap a 20gb f

Re: write a 20GB file

2010-05-15 Thread Nathan Rice
This is precisely the situation mmap was made for :) It has almost the same methods as a file so it should be an easy replacement. On Sat, May 15, 2010 at 10:05 AM, Nobody wrote: > On Fri, 14 May 2010 18:38:55 -0400, J wrote: > > >>> someone smarter than me can correct me, but file.write() will

Re: write a 20GB file

2010-05-15 Thread Nobody
On Fri, 14 May 2010 18:38:55 -0400, J wrote: >>> someone smarter than me can correct me, but file.write() will write when >>> it's buffer is filled, or close() or flush() are called. >> >> And, in all probability, seek() will either flush it immediately or cause >> the next write() to flush it bef

Re: write a 20GB file

2010-05-14 Thread Jackie Lee
Thanks to y'all. I should have be more careful reading the documentation. Cheers On Fri, May 14, 2010 at 10:07 PM, Martin v. Loewis wrote: >> The code works fine. I just don't know how f.write works. It says that >> file.write won't write the file until file.close or file.flush. > > You are misi

Re: write a 20GB file

2010-05-14 Thread J
On Fri, May 14, 2010 at 15:23, Nobody wrote: > On Fri, 14 May 2010 10:50:49 -0400, J wrote: > >> someone smarter than me can correct me, but file.write() will write when >> it's buffer is filled, or close() or flush() are called. > > And, in all probability, seek() will either flush it immediately

Re: write a 20GB file

2010-05-14 Thread Nobody
On Fri, 14 May 2010 10:50:49 -0400, J wrote: > someone smarter than me can correct me, but file.write() will write when > it's buffer is filled, or close() or flush() are called. And, in all probability, seek() will either flush it immediately or cause the next write() to flush it before writing

Re: write a 20GB file

2010-05-14 Thread Martin v. Loewis
> The code works fine. I just don't know how f.write works. It says that > file.write won't write the file until file.close or file.flush. You are misinterpreting the documentation. It certainly won't keep the entire file in memory. Instead, it has a fixed-size buffer (something like 8kiB or 32kiB

Re: write a 20GB file

2010-05-14 Thread J
On Fri, May 14, 2010 at 07:32, Jackie Lee wrote: > Thx, Dave, > > The code works fine. I just don't know how f.write works. It says that > file.write won't write the file until file.close or file.flush. So I > don't know if the following one is more efficient (sorry I forget to > add condition to

Re: write a 20GB file

2010-05-14 Thread Jackie Lee
Thx, Dave, The code works fine. I just don't know how f.write works. It says that file.write won't write the file until file.close or file.flush. So I don't know if the following one is more efficient (sorry I forget to add condition to break the loop): #! /usr/bin/env python #coding=utf-8 import

Re: write a 20GB file

2010-05-14 Thread Dave Angel
Jackie Lee wrote: Hello there, I have a 22 GB binary file, a want to change values of specific positions. Because of the volume of the file, I doubt my code a efficient one: #! /usr/bin/env python #coding=utf-8 import sys import struct try: f=open(sys.argv[1],'rb+') except (IOError,Exc

write a 20GB file

2010-05-14 Thread Jackie Lee
Hello there, I have a 22 GB binary file, a want to change values of specific positions. Because of the volume of the file, I doubt my code a efficient one: #! /usr/bin/env python #coding=utf-8 import sys import struct try: f=open(sys.argv[1],'rb+') except (IOError,Exception): print '