Sorry guys the segment is indeed fine, I was mislead because there was also a bug on my indexing filter which made the indexing fail and I did not check the segment but the index... So at least I did not loose all those correct fetches.
When I have checked the proposed fix for abort I will report status. 2009/8/19 MilleBii <[email protected]> > Well in the segment there is nothing but _temporary... and then after a > number of spinwaiting for the last 6 elements it aborts and the segment is > empty... > I guess everything is left in the tmp files. > If only it would abort gracefully and close the segment properly... > basically lost 200k URL. > > > 2009/8/19 Doğacan Güney <[email protected]> > > On Wed, Aug 19, 2009 at 09:40, MilleBii <[email protected]> wrote: >> >> > Quite frustrating problem , after 24h fetching properly >> > >> > fetcher is spinwaiting on a fetchqueue size of 6 and then goes in >> aborting >> > the fetch. >> > I have this problem from time to time and I have no idea why. >> > Can anyone help or suggest something to debug this issue. >> > >> >> When this happens, every URL successfully fetched should be stored in the >> segment. This part works for you right? Only those last 6 URLs are not >> fetched? >> >> >> > >> > -- >> > -MilleBii- >> > >> >> >> >> -- >> Doğacan Güney >> > > > > -- > -MilleBii- > -- -MilleBii-
