[
https://issues.apache.org/jira/browse/NUTCH-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dogacan Güney updated NUTCH-443:
--------------------------------
Attachment: NUTCH-443-draft-v3.patch
new patch, contains a possible fix for CrawlDbReducer problem.
This version finally works! (well, not really, but I can definitely say that it
almost kind of works..sometimes:)
I have two main issues with this patch:
1) If fetcher is in parsing mode, and parse returns a SUCCESS_REDIRECT,
fetcher handles this redirect. After this change, fetcher checks if the first
element of parseMap.values() (whatever that may be) has a SUCCESS_REDIRECT. It
is possible that a multi-entry parseMap has an parse element with a
SUCCESS_REDIRECT that is not the first element. (perhaps we can first check if
parseMap.get(originalUrl) returns a parse, if not use first element of
parseMap.values()? )
2) To be able to pass fetch time to not-actually-fetched-but-generated-in-parse
urls, I first put the original fetch time to content and then pass the value in
content to all elements in parseMap.values(). I guess this approach is not very
optimal since it passes fetch time around a lot.
> allow parsers to return multiple Parse object, this will speed up the rss
> parser
> --------------------------------------------------------------------------------
>
> Key: NUTCH-443
> URL: https://issues.apache.org/jira/browse/NUTCH-443
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 0.9.0
> Reporter: Renaud Richardet
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: NUTCH-443-draft-v1.patch, NUTCH-443-draft-v2.patch,
> NUTCH-443-draft-v3.patch, parse-map-core-draft-v1.patch,
> parse-map-core-untested.patch, parsers.diff
>
>
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser
> can return multiple parse objects, that will all be indexed separately.
> Advantage: no need to fetch all feed-items separately.
> see the discussion at
> http://www.nabble.com/RSS-fecter-and-index-individul-how-can-i-realize-this-function-tf3146271.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
