Robin Haswell wrote:
> On Fri, 2006-12-08 at 12:41 +0100, Andrzej Bialecki wrote:
>
>> Yes, most likely. Running complex regexes on hostile data, such as
>> unknown URLs, quite often ends up like this - that's why many
>> Internet-wide installations don't use regexes but combinations of
>> prefix/suffix/custom filters.. If you were running the fetcher in
>> non-parsing mode, this wouldn't happen during fetching but during
>> parsing - and you could've changed your config and restart just the
>> parsing, without refetching ... ah well.
>>
>> Anyway - it's most likely not hung, but runs very, very slowly. You
>> could give it a chance and let it run a few hours more, perhaps it will
>> go past these troublesome urls, and keep watching the size of temporary
>> data - if the files are not growing at all, then I'm afraid you will
>> have to kill the job, and avoid your boss for a couple of days ... :/
>>
>> (By the way, one can encounter most weird things in the wild ... I've
>> seen URLs that are several kilobytes long, containing all sorts of
>> illegal characters, containing nested unescaped URLs with invalid
>> protocols and so and so on ... so, when crawling Internet at large you
>> should be prepared for getting really nasty stuff. Complex regexes don't
>> cut it).
>>
>
>
> I see, thanks. Ah well. The scope my regex is simply
> glob("http://*.uk/*";). What filters would you recommend for doing this?
>
Prefix filter to cut off anything without "http://";. And then a
(non-existent) domain-suffix filter, which considers only domain
suffixes - this is easy to implement based on the suffix filter that
ships with Nutch.
> I'm guessing my use-case is pretty much the same as everyone else -
> people who want everything from a domain. Is it wise to ship with the
> regex urlfilter as the default filter?
>
> Anyway, any help would be great. I'll keep an eye on the temp data. If
> it rises I'll probably leave it going.
>
> Thanks
>
> -Rob
>
>
>
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
