On Mon, 5 Jan 2004, Marek Lewczuk wrote: > Dave Cramer wrote: > > connection failed :( > works for me... :-) (poland) >
We have small downtime because of upgrading server software, so this may be a reason for the problem. We're in stage of optimizing crawler because some sites are very-very ugly, for example, our crawler have discovered 2 millions URLs on http://ems-hitech.com/pgmanager/ ! 99.99 % of URLs are just 404 (document not found), but server does return 200 code )\:) So we have to explicitly exclude these pages. btw, archives.postgresql.org doesn't returns modification date in header. This prevent crawler to optimize downloading process. So, there are many problems, but we hope soon we'll tune crawling process. I estimate average time to refresh index about 1 week. > > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
