[
https://issues.apache.org/jira/browse/NUTCH-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrzej Bialecki updated NUTCH-443:
------------------------------------
Attachment: patch.txt
I'm not too happy with the direction you took in the latest patch. IMHO adding
crawl_parse to Indexer only complicates things, since you need to handle
additional conditions, not to mention the performance hit to process a lot more
data.
Instead, why don't we simply omit storing empty content in Fetcher / Fetcher2
when we get a redirect that we don't immediately follow? This would solve the
original issue.
Please see the patch to Fetcher2 that illustrates this issue.
> allow parsers to return multiple Parse object, this will speed up the rss
> parser
> --------------------------------------------------------------------------------
>
> Key: NUTCH-443
> URL: https://issues.apache.org/jira/browse/NUTCH-443
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 0.9.0
> Reporter: Renaud Richardet
> Assigned To: Chris A. Mattmann
> Priority: Minor
> Fix For: 1.0.0
>
> Attachments: NUTCH-443-draft-v1.patch, NUTCH-443-draft-v2.patch,
> NUTCH-443-draft-v3.patch, NUTCH-443-draft-v4.patch, NUTCH-443-draft-v5.patch,
> NUTCH-443-draft-v6.patch, NUTCH-443-draft-v7.patch,
> NUTCH-443.022507.patch.txt, NUTCH-443.02282007-v2.patch,
> NUTCH-443.02282007.patch, NUTCH-443.08052007.patch,
> parse-map-core-draft-v1.patch, parse-map-core-untested.patch, parsers.diff,
> patch.txt, redirect_and_index.patch
>
>
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser
> can return multiple parse objects, that will all be indexed separately.
> Advantage: no need to fetch all feed-items separately.
> see the discussion at
> http://www.nabble.com/RSS-fecter-and-index-individul-how-can-i-realize-this-function-tf3146271.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
