wuqi wrote:
I am also trying to improve the Generator efficiency. The current Generator all
the URLs in crawlDB are dumped out and ordered during the map process and the
reduce process will try to find top N pages or maxPerhost page for you. If the
page amounts in the CrawlDB is much bigger than N, Need all the page be dumped
out during map process? We may just need to provide (2~3)*N pages during the
map process,and then reduce select N pages from dumped out (2~3)n pages. this
might improve the Generator efficiency ..
Yes, the generate process will be faster. But of course less accurate.
And if you're working with generate.max.per.host, then it is likely that
your segment will be less than topN in size.
I think maybe the crawlDB can be stored based on two layers, the first layer is
Host,the second layer is pageURL.This can improve efficiency when using max
pages per host to generator fetch list.
My first thought is that such an approach makes it hard to select the
best scoring urls.
Perhaps we could design the process in such way that some intermediate
results like the part of the crawldb which is sorted during generation
(this contains all urls elegible for fetching) are saved and reused. Why
sort everything again each time when you know only a fraction of the
urls have been updated?
Mathijs
Hbase can greatly improve the updateDB efficiency,because no need to dump all
URLS in crawldb, it just need to append a new column with DB_Fetched for the
URL fetched. The other benefit brought by Hbase is that we can easily change
schema of crawlDB for example add IP address for each URL... I am not familiar
with how the HBase behavior under the interface.. so selecting out might be
problem...
----- Original Message -----
From: "Mathijs Homminga" <[EMAIL PROTECTED]>
To: <nutch-dev@lucene.apache.org>
Sent: Wednesday, May 07, 2008 6:28 AM
Subject: Internet crawl: CrawlDb getting big!
Hi all,
The time needed to do a generate and an updatedb depends linearly on the
size of the CrawlDb.
Our CrawlDb currently contains about 1.5 billion urls (some fetched, but
most of them unfetched).
We are using Nutch 0.9 on a 15-node cluster. These are the times needed
for these jobs:
generate: 8-10 hours
updatedb: 8-10 hours
Our fetch job takes about 30 hours, in which we fetch and parse about 8
million docs (limited by our current bandwidth).
So, we spent about 40% of our time on CrawlDb administration.
The first problem for us was that we didn't make the best use of our
bandwidth (40% of the time no fetching). We solved this by designing a
system which looks a bit like the FetchCycleOverlap
(http://wiki.apache.org/nutch/FetchCycleOverlap) recently suggested by Otis.
Another problem is that as the CrawlDb grows, the admin time increases.
One way to solve this is by increasing the topN each time so the ratio
between admin jobs and the fetch job remains constant. However, we will
end up with extreme long cycles and large segments. Some of this we
solved by generating multiple segments in one generate job and only
perform an updatedb when (almost) all of these segments are fetched.
But still. The number of urls we select (generate), and the number of
urls we update (updatedb) is very small compared to the size of the
CrawlDb. We were wondering if there is a way such that we don't need to
read in the whole CrawlDb each time.
How about putting the CrawlDb in HBase? Sorting (generate) might become
a problem then...
Is this issue addressed in the Nutch2Architecture?
I'm happily willing to spend some more time on this, so all ideas are
welcome.
Thanks,
Mathijs Homminga
--
Knowlogy
Helperpark 290 C
9723 ZA Groningen
The Netherlands
+31 (0)50 2103567
http://www.knowlogy.nl
[EMAIL PROTECTED]
+31 (0)6 15312977
>
--
Knowlogy
Helperpark 290 C
9723 ZA Groningen
+31 (0)50 2103567
http://www.knowlogy.nl
[EMAIL PROTECTED]
+31 (0)6 15312977
