On 1 Nov, 22:25, Orest Kozyar <[EMAIL PROTECTED]> wrote: > > > I'm working on a CGI script that pulls XML data from a public database > > > Ah, I missed that bit on first read. Consider using something different than > > CGI here if you want to do caching. FCGI would allow you to do in-memory > > caching, for example, as would mod_python and a lot of other solutions. > > What I'm aiming for is sort of a "permanent" disk cache/mirror that > adds records as needed. The main issue is that the database (PubMed) > requests that we limit requests to once every three seconds.
Drifting off-topic slightly, it is possible to download archives of PubMed, but you'll need a lot of disk space; you probably know this already. Putting the data into something rapidly searchable can demand even more space, depending on what you use and what you want to be able to search for. I guess it depends on whether you're retrieving relatively few documents across the whole of PubMed or whether your searches are concentrated in particular sections of the whole archive. Paul -- http://mail.python.org/mailman/listinfo/python-list
