On Fre, 07 Jan 2000, Philippe CROCHAT wrote:
> And it seems that my program stops while fetching some url for a very very
> long time (some hours sometimes)
I've usually seen this when the UserAgent tries to connect to a new site
(i.e. open a new socket). Once it's connected, the download should be
correctly timing out if no data has been read for TIMEOUT seconds. The only
solution I could think of would be to use "alarm" when opening the
IO::Socket.
> Moreover i got "broken pipe" when i register too much urls (around 1000
> urls)
Mmh, your settings (max_hosts = 8, max_req = 8) don't look like they're
stressing the OS too much. Usually Parallel::UserAgent should only open at
most 8x8=64 connections this way. Registering more should only result in
adding them into a internal queue - no additional connections are opened.
Maybe it's just coincidence that some broken sites are among those 1000
urls? Have you been able to reproduce the problem?
-m
--
Marc Langheinrich
[EMAIL PROTECTED]
