Oops continuing previous mail. So I wonder if there would be a better algorithm 'generate' which would maintain a constant rate of host per 100 url ... Below a certain threshold it stops or better starts including URLs of lower scores.
Using scores is de-optimzing the fetching process... Having said that I should first read the code and try to understand it. 2009/12/3, MilleBii <[email protected]>: > Observing my fetch cycles perf. It looks like there is always a rather > long tail. > I saw it on 10k, 150k, 450k fetch runs. > > Of course you can cut-off the tail with the patch 770 made by Julien > (thx), I did some dry test looks like working, so I'm going to move it > to production. > > Yet, what seems to make the difference is the good mix of URL, ie nbr > of different host per 100/ URL. > > > -- > -MilleBii- > -- -MilleBii-
