On Sat, 2013-01-05 at 22:11 +0000, sebb wrote: > On 5 January 2013 21:33, vigna <[email protected]> wrote: > >> But why would you want a web crawler to have 10-20K simultaneously > >> opened connections in the first place? > > > > (I thought I answered this, but it's not on the archive. Boh.) > > > > Having a few thousands connection open is the only way to retrieve data > > respecting politeness (e.g., not banging the same site too often). > > Huh? > There are surely other ways to achieve that goal. >
I could not agree more. I personally think that closing idle connections and letting the server reclaim the resources associated with them (potentially enabling the server to serve other clients) would be more 'polite'. It is cheaper for both the client and the server to close connections more frequently than keeping them alive just in case. Oleg --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
