Phlip <phlip2...@gmail.com> writes: > If the file were huge, the file.read() would allocate a big string and > thrash memory. (Yes, in 2011 that's still a problem, because these > files could be movies and whatnot.)
I did a crc32 calculator like that and actually ran into some kind of string length limit with large files. So I switched to 4k blocks and the speed is about the same as a C implementation in the program cksfv. Well, of course crc32 is usually done with a table lookup, so it's always fast. I just picked 4k, since it's the page size in x86 systems and also a common block size for file systems. Seems to be big enough. io.DEFAULT_BUFFER_SIZE is 8k here. I suppose using that would be the proper way. -- http://mail.python.org/mailman/listinfo/python-list
