[
https://issues.apache.org/jira/browse/NUTCH-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771625#action_12771625
]
David Stuart commented on NUTCH-585:
------------------------------------
Hi Andrea,
I hope your week of demo's went well. I to would be interested in this code as
I would like to look at extending to it be slightly more generic allowing for
regular expression matches or an xpath like model (the plan is still
formulating). From the web crawler view it would be a hard one to get right but
we have about 26 sites that are will know to us that we wish to crawl and have
common blocks that we wish to remove which a configurable version of your code
may achieve.
Look forward to see your patch
Regards,
David Stuart
> [PARSE-HTML plugin] Block certain parts of HTML code from being indexed
> -----------------------------------------------------------------------
>
> Key: NUTCH-585
> URL: https://issues.apache.org/jira/browse/NUTCH-585
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Environment: All operating systems
> Reporter: Andrea Spinelli
> Priority: Minor
>
> We are using nutch to index our own web sites; we would like not to index
> certain parts of our pages, because we know they are not relevant (for
> instance, there are several links to change the background color) and
> generate spurious matches.
> We have modified the plugin so that it ignores HTML code between certain HTML
> comments, like
> <!-- START-IGNORE -->
> ... ignored part ...
> <!-- STOP-IGNORE -->
> We feel this might be useful to someone else, maybe factorizing the comment
> strings as constants in the configuration files (say parser.html.ignore.start
> and parser.html.ignore.stop in nutch-site.xml).
> We are almost ready to contribute our code snippet. Looking forward for any
> expression of interest - or for an explanation why waht we are doing is
> plain wrong!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
