On Nov 28, 2009, at 12:42am, Eran Zinman wrote:
Thanks for your help MillBii!
I will definitely try the squareroot option - but is that only valid
for
outlinks or also affects pages linking to the page?
Did you try implementing automatic Regex generation? I'm doing focused
crawling but I'm also thinking about scaling it in the future.
Also I will be happy to know if anyone else have any other
suggestion (or
already implemented strategy) - I think this issue affects most of
the Nutch
community - at least people that use Nutch for focused crawling.
I believe I wrote several emails to this list a while ago about how we
handled this for Krugle, so those could prove useful.
The basic concept was:
- Use machine learning to train a classifier (we used term vector
similarity and max entropy) for good & bad pages.
- Run this on each fetched page, and use the results to adjust the
outbound OPIC link scores.
We didn't bother following backlinks. I don't think this would impact
your crawl frontier generation much.
But a different approach would be to stop the crawl after some
significant # of pages had been fetched, then use a modified version
of the PageRank calculation that factors in the classification score.
This could give you better static stores, and you could also use this
new score to recalculate scores for unfetched pages, based on the
source pages' static scores.
-- Ken
Thanks,
Eran
On Fri, Nov 27, 2009 at 8:29 PM, MilleBii <[email protected]> wrote:
Well I have created for my own application is topical-scoring
plugin :
1. first I needed to score the pages after parsing based on my
regular
expression
2. then I searched several options on to how boost score of that
pages... I
have only found a way to boost the score of the outlinks of these
pages
that
have content which I wanted. Not perfect but so be it there is a high
likelyhood in my case that adjacent pages have also content which I
want.
3. then how to boost the score... this took me a while to figure
out, I
leave you all the options I tried. The good comprise I found is the
following:
if the page has content I want and score < 1.0f than score=
squareroot(score)... in this way you are adding weight to the pages
which
have content you are looking (since score is usually below 1.
squareroot(x)
is bigger than x).
Of course there are some down side to that approach, it is more
difficult
to
get the crawler to go outsides sites that have content your are
looking
for,
it is a bit like digging a hole and until you have finished the
hole it
will
get the crawler to explore it... experimentally I have found that
it works
nicely for me though, if you limit the nbre of URLS per site it
won't spend
it's life on them.
We could try to generalize this plug-in by putting the regular
expression
as
as config item because that is really the only thing which is
specific to
my
application I believe.
2009/11/27 Eran Zinman <[email protected]>
Hi all,
I'm try to figure out ways to improve Nutch focused crawling
efficiency.
I'm looking for certain pages inside each domain which contains
content
I'm
looking for.
I'm unable to know that a certain URL contains what I'm looking for
unless
I
parse it and do some analysis on it.
Basically I was thinking about two methods to improve crawling
efficiency:
1) Whenever a page is found which contains the data I'm looking for,
improve
overall score for all pages linking to it (and pages linking to
them and
so
on...), assuming they have other links that point to content I'm
looking
for.
2) Once I already found several pages that contain relevant data -
create
a
Regex automatically to match new urls which might contain usable
content.
I've started to read about the OPIC-score plugin but was unable to
understand if it can help me or not with issue no. 1.
Any idea guys? I will be very grateful for any help or things that
can
point
me in the right direction.
Thanks,
Eran
--
-MilleBii-
--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
