It sounds like you will have many regular files (i.e., random access/seekable inputs as opposed to things like Unix pipes). On Linux with glibc, memfiles.open is probably the fastest approach which uses memchr internally to find line boundaries. E.g. (right from memfiles documentation), import memfiles for line in lines(memfiles.open("foo")): inc(i)
Your mileage on this may vary from OS to OS or libc to libc. I have no idea which if any Microsoft/Windows versions have well-optimized libc memchr() implementations.