[ https://issues.apache.org/jira/browse/NUTCH-585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
N. Hira updated NUTCH-585: -------------------------- Attachment: nutch-585-jostens-excludeDIVs.patch We use Solr/Nutch on our corporate web site and are very happy with the results. Thank you. We have struggled with something similar to NUTCH-585 for a few months now. Although it is different from the original intent, here's a quick/short patch that might help get this feature going again. h4.Intended use: - Let's assume you're crawling a set of internal web sites and would like to exclude certain HTML fragments (from indexing) like the navigation and other common content. - If these fragments are contained in DIVs with IDs like "menuNav", "footerNav", etc., then you can now add a new property to nutch-site.xml to exclude these DIVs. - If you don't set this property, the normal behavior remains (backward compatible) {code:xml} <property> <name>parser.html.divIDsToExclude</name <value>account_menu_container,footer_menu_container,legal,main_menu_container</value> <description> A comma-delimited list of DIV IDs whose content will not be indexed. Use this to tell the HTML parser to ignore, for example, site navigation text. Note that DIVs with these IDs, and their children, will be silently ignored by the parser so verify the indexed content with Luke to confirm results. </description> </property> {code} h4.Inclusion/growth: - This code was written against nutch 1.2 and is backward compatible in that the new behavior is only present if configured. - In future, it might be good to have different "strategy patterns" for how exclusions are determined; some might need algorithmic detection (whole web crawls), others might prefer jquery-selectors for HTML fragments, etc. Best regards, -h Hira, N.R. (Jostens, Inc.) > [PARSE-HTML plugin] Block certain parts of HTML code from being indexed > ----------------------------------------------------------------------- > > Key: NUTCH-585 > URL: https://issues.apache.org/jira/browse/NUTCH-585 > Project: Nutch > Issue Type: Improvement > Affects Versions: 0.9.0 > Environment: All operating systems > Reporter: Andrea Spinelli > Priority: Minor > Attachments: nutch-585-jostens-excludeDIVs.patch > > > We are using nutch to index our own web sites; we would like not to index > certain parts of our pages, because we know they are not relevant (for > instance, there are several links to change the background color) and > generate spurious matches. > We have modified the plugin so that it ignores HTML code between certain HTML > comments, like > <!-- START-IGNORE --> > ... ignored part ... > <!-- STOP-IGNORE --> > We feel this might be useful to someone else, maybe factorizing the comment > strings as constants in the configuration files (say parser.html.ignore.start > and parser.html.ignore.stop in nutch-site.xml). > We are almost ready to contribute our code snippet. Looking forward for any > expression of interest - or for an explanation why waht we are doing is > plain wrong! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.