[ http://issues.apache.org/jira/browse/NUTCH-240?page=comments#action_12372574 ]
Doug Cutting commented on NUTCH-240: ------------------------------------ First, I hope my critical remarks were not taken personally. I am thankful for this and all of your contributions. > Initially, I did as you suggest, i.e. I created a method to calculate one > float value for the purpose of selecting topN. However, I wanted to avoid > changing CrawlDatum.compareTo - if we put ScoringFilters there, it would be a > big performance hit. OTOH, if we overwrite the primitive float in > CrawlDatum.score it seemed to me we should store its earlier value, and then > possibl restore - as the value for selecting topN may have nothing to do with > the "real" score. In Generate.java, can't we just change the key type in the first pass to be a FloatWritable holding the score, and the value to be <CrawlDatum,Url>? Then we'd never alter the CrawlDatum and there'd be no need to restore it. > passScoreBeforeParsing/passScoreAfterParsing: again, I agree it looks > strange, but that's what we do at the moment, I just extracted it into an > interface. I'd love to skip this altogether, if there is a way. I think we should spend a little more time thinking about how to make this a nice API before we start having folks implement it. Once an interface is added, it's much harder to change. I don't have much time to spend on this today, but might next week. > Scoring API: extension point, scoring filters and an OPIC plugin > ---------------------------------------------------------------- > > Key: NUTCH-240 > URL: http://issues.apache.org/jira/browse/NUTCH-240 > Project: Nutch > Type: Improvement > Versions: 0.8-dev > Reporter: Andrzej Bialecki > Attachments: patch.txt > > This patch refactors all places where Nutch manipulates page scores, into a > plugin-based API. Using this API it's possible to implement different scoring > algorithms. It is also much easier to understand how scoring works. > Multiple scoring plugins can be run in sequence, in a manner similar to > URLFilters. > Included is also an OPICScoringFilter plugin, which contains the current > implementation of the scoring algorithm. Together with the scoring API it > provides a fully backward-compatible scoring. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
