[
https://issues.apache.org/jira/browse/DROIDS-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingfai Ma updated DROIDS-47:
-----------------------------
Attachment: DROIDS-47.patch
DROIDS-47_profile-after.png
be honest, i think it does not make sense to do a size checking that throw an
exception.
when a crawling job is being run, links are keep adding to the queue. it make
sense to have a limit to throttle memory usage, so one may use a bounding queue
and additional links cannot be added before links are processed. However,
throwing exception means the extractor thread will get exception, the links
will be discarded, and either the extractor thread fails or the whole Droids
job fails. (hope it would be the former case)
my patch does the following:
- commented the queue size checking logic
- added a constructor to allow user to supply a Queue. if anyone need a queue
that throw exception when it is full, he may supply one.
- commented the testMaxSize method in TestSimpleQueue, you may want to delete
the content instead
> SimpleTaskQueue is very slow to init when there is a large number of seed URLs
> ------------------------------------------------------------------------------
>
> Key: DROIDS-47
> URL: https://issues.apache.org/jira/browse/DROIDS-47
> Project: Droids
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.01
> Reporter: Mingfai Ma
> Attachments: DROIDS-47.patch, DROIDS-47_profile-after.png,
> DROIDS-47_profile.png
>
>
> when i put 300,000 seeds url to CrawlingDroid with SimpleTaskQueueWithHistory
> and init() it, it takes an hour to complete on my Quad core server.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
