Re: Slowdown when fetching multiple records

[Tony Bowden]

On Mon, Feb 20, 2006 at 09:23:00PM -0600, Bryan Baldus wrote:
 I've not used Perl-based Z39.50 searching, but I think LC requires a 
 6-second pause between searches, to reduce burden on their servers. 
 While searching using MarcEdit 4.6 (or pre-5), we were locked out for 
 the day for not following this new (Oct. 2005?) rule. MarcEdit 5 beta 
 allows a 6-second pause for servers that might require it, like LC's.
 I don't know how the Perl modules handle this pause for batch 
 searching LC, but it might be why you are experiencing a delay.

Thanks for the tip. I've run into this problem as well. I sleep for a
while between requests now, but it's useful to know that it should be at
least 6 seconds. Is there somewhere on their site that documents this? I
spent a while looking last week but couldn't find it.

However, I think the slowdown I was getting is a different problem. I
don't know enough about how Z39.50 servers actually work, but I think
that requesting each book in turn just had a high overhead of network
traffic, and presumably server load and/or queuing of some sort.
Pre-fetching batches of 20 made everything much much quicker - even
including a sleep inbetween each successive fetch.

Thanks,

Tony

Re: Slowdown when fetching multiple records

[Ashley Sanders]

Tony Bowden wrote:


However, I think the slowdown I was getting is a different problem. I
don't know enough about how Z39.50 servers actually work, but I think
that requesting each book in turn just had a high overhead of network
traffic,


Correct. If you request a total of ten records one at a time
then that equates to ten request/response conversations between
client and server. This realy is very wasteful of network and
server resources. It is for reasons of efficiency that z39.50
allows you to ask for multiple records from the server in
one fell swoop. (A server may limit the number of records
it will return in any one reuqest, but I wouldn't have thought
20 records would cause any server a problem (unless they are
very large records.))

Ashley.
--
Ashley Sanders [EMAIL PROTECTED]
Copac http://copac.ac.uk -- A MIMAS service funded by JISC