[
https://issues.apache.org/jira/browse/NUTCH-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609282#comment-14609282
]
Sebastian Nagel commented on NUTCH-2038:
----------------------------------------
Hi [~asitang], I was able to reproduce the exception. To train the classifier a
MapReduce job is launched:
* it obviously does not have the classes of the plugin at hand. Each plugin
uses its own class loader (see
[[1|http://wiki.apache.org/nutch/WhatsTheProblemWithPluginsAndClass-loading]]).
Don't know whether it's possible to make the plugin classes available to the
training job.
* if the classifier is trained inside the parse step of a crawl, this will mean
that a job/task launches another job. Sounds awkward. Again: I don't know
whether this will work at all in local and in distributed mode.
Sorry, that I haven't seen this dependency on running a MapReduce job before.
Unfortunately, Mahout does not provide a non-MapReduce version of Naive Bayes
([[2|https://mahout.apache.org/users/basics/algorithms.html]]). Needs some
thoughts to get a solution. In doubt, the training step could be run separately
beforehand.
> Naive Bayes classifier based html Parse filter (for filtering outlinks)
> -----------------------------------------------------------------------
>
> Key: NUTCH-2038
> URL: https://issues.apache.org/jira/browse/NUTCH-2038
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, injector, parser
> Reporter: Asitang Mishra
> Assignee: Chris A. Mattmann
> Labels: memex, nutch
> Fix For: 1.11
>
>
> A html parse filter that will filter out the outlinks in two stages.
> Classify the parse text and decide if the parent page is relevant. If
> relevant then don't filter the outlinks. If irrelevant then go thru each
> outlink and see if the url contains any of the important words from a list.
> If it does then let it pass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
