[jira] Updated: (NUTCH-546) file URL are filtered out by the crawler
[ https://issues.apache.org/jira/browse/NUTCH-546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doğacan Güney updated NUTCH-546: Attachment: NUTCH-546-validator-plugin_v1.patch Here is a patch that removes UrlValidator code from nutch code and adds it as a plugin (urlfilter-validator). file URL are filtered out by the crawler Key: NUTCH-546 URL: https://issues.apache.org/jira/browse/NUTCH-546 Project: Nutch Issue Type: Bug Components: fetcher Affects Versions: 1.0.0 Environment: Windows XP Nutch trunk from Monday, August 20th 2007 Reporter: Marc Brette Assignee: Doğacan Güney Attachments: NUTCH-546-validator-plugin_v1.patch, NUTCH-546.patch I tried to index file system using the file:/ protocol, which worked fine in version 0.9 The file URL are being filtered out and not fetched at all. I investigated the code and saw that there are 2 issues: 1) One is with the class UrlValidator: when validating an URL, it check the 'authority', a combination of host and port. As it is null for file, the URL is rejected. 2) Once this check is removed, files that contain space characters (and maybe other characters to be URL encoded) are also filtered out. It maybe be because the file protocol plugin doesn't URL encode space characters and/or UrlValidator is enforce the rule to encode such character. To workaround these issues, I just commented out UrlValidator checks and it works fine. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-524) Generate Problem with Single Node
[ https://issues.apache.org/jira/browse/NUTCH-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525419 ] Doğacan Güney commented on NUTCH-524: - Hi Ian and Daniel, Have you tried max.threads.per.host option? Or are you still working on this one? Generate Problem with Single Node - Key: NUTCH-524 URL: https://issues.apache.org/jira/browse/NUTCH-524 Project: Nutch Issue Type: Bug Components: generator Affects Versions: 0.9.0 Environment: All Reporter: Daniel Clark Priority: Minor Fix For: 0.9.0 Attachments: nutch-0.9_PartitionUrlByHost.patch Nutch with Hadoop has problems with a single node in URL list when there is a cluster of two or more machines. I will provide a fix for this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-548) Move URLNormalizer from Outlink to ParseOutputFormat
[ https://issues.apache.org/jira/browse/NUTCH-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525452 ] Emmanuel Joke commented on NUTCH-548: - My mistake, you re right i was using the command crawl to make my test, and i didn't noticed that within the code it defined the urlfiter and urnormalizer to TRUE. Anyway, this current patch is still valid and useful. Thanks again for those explanation. Move URLNormalizer from Outlink to ParseOutputFormat Key: NUTCH-548 URL: https://issues.apache.org/jira/browse/NUTCH-548 Project: Nutch Issue Type: Improvement Components: fetcher Reporter: Emmanuel Joke Assignee: Emmanuel Joke Priority: Minor Fix For: 1.0.0 Attachments: NUTCH-548.patch The idea is to avoid instantiating a new URLNormalizer for every OutLink. So I move this operation to the ParseOutputFormat object. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-530) Add a combiner to improve performance on updatedb
[ https://issues.apache.org/jira/browse/NUTCH-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525475 ] Andrzej Bialecki commented on NUTCH-530: - I'm still against this patch, exactly because we are not sure how many times the ScoringFilters will be executed - it may be once, twice or N times. The current contract for ScoringFilters is that they are executed once. CrawlDbReducer itself does not reduce all inlinked datums to a single CrawlDatum - it's up to the scoring filters to do whatever they want to do with all inlinks - although it's true that scoring-opic performs an operation equivalent to this, this may not always be the case. Second, let's consider the following scenario (BTW, this is close to one of the ScoringFilters that I actually implemented, so it's not far fetched): let's say I implemented a ScoringFilter that checks for existence of a flag in CrawlDatum (presumably put there by Generator), and based on the value of this flag it counts the score from inlinks differently. Then it clears the flag to mark a successful update. If we ran updatedb that includes your patch, this operation would work ok in the first spill from the Combiner (although with vastly incomplete information), and then it would fail to do the right thing on subsequent runs through the Combiner or Reducer, because the flag would be already reset. Add a combiner to improve performance on updatedb - Key: NUTCH-530 URL: https://issues.apache.org/jira/browse/NUTCH-530 Project: Nutch Issue Type: Improvement Environment: java 1.6 Reporter: Emmanuel Joke Assignee: Emmanuel Joke Fix For: 1.0.0 Attachments: NUTCH-530.patch We have a lot of similar links with status linked generated at the ouput of the map task when we try to update the crawldb based on the segment fetched. We can use a combiner to improve the performance. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Labeling URLs a-la Google
Hello everybody, I'm working on a project that is essentially a searchable database for academic citations at the University of Pittsburgh. One of our searching requirements was to be able to break the search results into sections--in order to do this, I implemented something similar to Google's labels. It's based heavily on the example plugin, and maybe not so pretty code-wise, but it's a start. Downloadable here: http://upclose.lrdc.pitt.edu/people/maki_assets/nutch-regex-label.tar.gz You configure it by adding something like the below to your nutch-site.xml file: property nameextension.regexlabeler.labels/name value http://dev3\.informalscience\.org/research.*\.php.* = firsttag secondtag thirdtag, http://dev3\.informalscience\.org/project.*\.php.* = project, http://www.?\.informalscience\.org.* = oldsite, http://dev3\.informalscience\.org.* = devsite /value /property Notes: * Format of each line is regular expression=labels, space delimited * URLS must be unique. * Multiple tags for the same pattern are delimited by a space. Hope this saves somebody some time, -Jeff (BTW, Nutch as worked very well for us--excellent project!)
Limiting outlink tags.
Hi, I have noticed that Nutch considers img/@src as an outlink. I suppose in many cases people do not want to threat image as an outlink. At least I don't want. The same case is with script/@src. But, it seems there is no way to limit outlink tags. The DOMContentUtils.getOutlinks() takes links from all a,area,form,frame,iframe,script,link,img. Only form element can be turned off by parser.html.form.use_action parameter. I would suggest to introduce a new configuration parameter which could be used to turn on or off certain elements. It could be simply done by single parameter, which would contain coma separated list of tags to be turned off. What is your opinion? If you think it is a valid issue I can make a patch for this. Regards, Marcin
[jira] Created: (NUTCH-549) Bug
Bug --- Key: NUTCH-549 URL: https://issues.apache.org/jira/browse/NUTCH-549 Project: Nutch Issue Type: Bug Reporter: crossany -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
