fast text processing

2006-02-21 Thread Alexis Gallagher
(I tried to post this yesterday but I think my ISP ate it. Apologies if this is a double-post.) Is it possible to do very fast string processing in python? My bioinformatics application needs to scan very large ASCII files (80GB+), compare adjacent lines, and conditionally do some further

Re: fast text processing

2006-02-21 Thread Alexis Gallagher
Steve, First, many thanks! Steve Holden wrote: Alexis Gallagher wrote: filehandle = open(data,'r',buffering=1000) This buffer size seems, shall we say, unadventurous? It's likely to slow things down considerably, since the filesystem is probably going to naturally wnt to use a rather