Re: Python 2.5, problems reading large ( 4Gbyes) files on win2k
On 3/4/07, Paul Duffy [EMAIL PROTECTED] wrote: Bill Tydeman wrote: Just curious, but since the file size limitation on NTFS is 4 GB, have you confirmed that it isn't some other part of the interaction that is causing the problem? What FS is hosting the files? I don't think that is correct. Groovy version of app runs just fine.' That should have been pre-NTFS (i.e. FAT32) but I have had problems with files larger than 4GB on NTFS. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.5, problems reading large ( 4Gbyes) files on win2k
Bill Tydeman wrote: Just curious, but since the file size limitation on NTFS is 4 GB, have you confirmed that it isn't some other part of the interaction that is causing the problem? What FS is hosting the files? I don't think that is correct. Groovy version of app runs just fine. On 2 Mar 2007 10:09:15 -0800, [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Folks, I've a Python 2.5 app running on 32 bit Win 2k SP4 (NTFS volume). Reading a file of 13 GBytes, one line at a time. It appears that, once the read line passes the 4 GByte boundary, I am getting occasional random line concatenations. Input file is confirmed good via UltraEdit. Groovy version of the same app runs fine. Any ideas? Cheers -- http://mail.python.org/mailman/listinfo/python-list -- There is no reason for any individual to have a computer in his home. Ken Olsen, President, Digital Equipment, 1977 US computer engineer industrialist (1926 - ) -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.5, problems reading large ( 4Gbyes) files on win2k
On Mar 2, 10:09 am, [EMAIL PROTECTED] wrote: Folks, I've a Python 2.5 app running on 32 bit Win 2k SP4 (NTFS volume). Reading a file of 13 GBytes, one line at a time. It appears that, once the read line passes the 4 GByte boundary, I am getting occasional random line concatenations. Input file is confirmed good via UltraEdit. Groovy version of the same app runs fine. Any ideas? Cheers It appears to be a bug. I am able to reproduce the problem with the code fragment below. It creates a 12GB file with line lengths ranging from 0 to 126 bytes, and repeating that set of lines 150 times. It fails on W2K SP4 with both Python 2.4 and 2.5. It works correctly on Linux (Ubuntu 6.10). I have reported on SourceForge as bug 1672853. # Read and write a huge file. import sys def write_file(end = 126, loops = 150, fname='bigfile'): fh = open(fname, 'w') buff = 'A' * end for k in range(loops): for t in range(end+1): fh.write(buff[:t]+'\n') fh.close() def read_file(end = 126, fname = 'bigfile'): fh = open(fname, 'r') offset = 0 loops = 0 for rec in fh: if offset != len(rec.strip()): print 'Error at loop:', loops print 'Expected record length:', offset print 'Actual record length:', len(rec.strip()) sys.exit(0) offset += 1 if offset end: offset = 0 loops += 1 if not loops % 1: print loops fh.close() if __name__ == '__main__': write_file(loops=150) read_file() casevh -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.5, problems reading large ( 4Gbyes) files on win2k
Just curious, but since the file size limitation on NTFS is 4 GB, have you confirmed that it isn't some other part of the interaction that is causing the problem? What FS is hosting the files? On 2 Mar 2007 10:09:15 -0800, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Folks, I've a Python 2.5 app running on 32 bit Win 2k SP4 (NTFS volume). Reading a file of 13 GBytes, one line at a time. It appears that, once the read line passes the 4 GByte boundary, I am getting occasional random line concatenations. Input file is confirmed good via UltraEdit. Groovy version of the same app runs fine. Any ideas? Cheers -- http://mail.python.org/mailman/listinfo/python-list -- There is no reason for any individual to have a computer in his home. Ken Olsen, President, Digital Equipment, 1977 US computer engineer industrialist (1926 - ) -- http://mail.python.org/mailman/listinfo/python-list
Python 2.5, problems reading large ( 4Gbyes) files on win2k
Folks, I've a Python 2.5 app running on 32 bit Win 2k SP4 (NTFS volume). Reading a file of 13 GBytes, one line at a time. It appears that, once the read line passes the 4 GByte boundary, I am getting occasional random line concatenations. Input file is confirmed good via UltraEdit. Groovy version of the same app runs fine. Any ideas? Cheers -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.5, problems reading large ( 4Gbyes) files on win2k
[EMAIL PROTECTED] wrote: I've a Python 2.5 app running on 32 bit Win 2k SP4 (NTFS volume). Reading a file of 13 GBytes, one line at a time. It appears that, once the read line passes the 4 GByte boundary, I am getting occasional random line concatenations. Input file is confirmed good via UltraEdit. Groovy version of the same app runs fine. Any ideas? Do you open the file in universal newline mode -- open(filename, U) --, and if not, does the problem persist if you do? Peter -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.5, problems reading large ( 4Gbyes) files on win2k
I am not using the universal newline. File reading loop is essentially... ifile = open(fileName, r) for line in ifile ... Thanks Peter Otten wrote: [EMAIL PROTECTED] wrote: I've a Python 2.5 app running on 32 bit Win 2k SP4 (NTFS volume). Reading a file of 13 GBytes, one line at a time. It appears that, once the read line passes the 4 GByte boundary, I am getting occasional random line concatenations. Input file is confirmed good via UltraEdit. Groovy version of the same app runs fine. Any ideas? Do you open the file in universal newline mode -- open(filename, U) --, and if not, does the problem persist if you do? Peter -- http://mail.python.org/mailman/listinfo/python-list
