I'm a bit confused on where the requirement to *crawl* these sites comes into it? From what you're saying it looks like you're just talking about parsing the contents of a list of sites that you're trying to extract data from. In which case there's not much of a use case for Nutch... or am I confused?
On Tue, Jun 11, 2013 at 1:26 PM, Tony Mullins <tonymullins...@gmail.com>wrote: > Yes all the web pages will have different HTML structure/layout and I would > have to identify/define a XPath expression for each one of them. > > But I am trying to come up with generic output format for these XPath > expressions.... so whatever the XPath expression is I want to have result > in lets say Field A , Field B , Field C . In some cases some of these > fields could be blank as well. So I could map them to my Solr schema > properly. > > In this regard I was hopping to get some help or guideline from your past > experiences ... > > Thanks, > Tony > > > On Tue, Jun 11, 2013 at 10:09 PM, AC Nutch <acnu...@gmail.com> wrote: > > > Hi Tony, > > > > So if I understand correctly, you have 100+ web pages, each with a > totally > > different format that you're trying to extract separate/unrelated pieces > of > > information from. If there's no connection between any of the web pages > or > > any of the pieces of information that you're trying to extract then it's > > pretty much unavoidable to have to provide separate identifiers and cases > > for finding each one. Markus' suggestion I believe is to just have a > > "dictionary" file with URL as the key and XPath expression for the info > > that you want as the value. No matter what crawling/parsing platform > you're > > using a solution of that sort is pretty much unavoidable with the > > assumptions given. > > > > That being said, is there any common form that the data you're trying to > > extract from these pages follows? Is there a regex that could match it or > > anything else that might identify it in a common way? > > > > Alex > > > > > > On Tue, Jun 11, 2013 at 12:59 PM, Tony Mullins <tonymullins...@gmail.com > > >wrote: > > > > > Hi Markus, > > > > > > I couldn't understand how can I avoid switch cases in your suggested > > > idea.... > > > > > > I would have one plugin which will implement HtmlParseFilter and I > would > > > have to check the current URL by getting content.getUrl() and this all > > will > > > be happening in same class so I would have to add swicth cases... I may > > > could add xpath expression for each site in separate files but to get > > XPath > > > expression I would have to decide which file I have to read and for > that > > I > > > would have to add my this code logic in swith case.... > > > > > > Please correct me if I am getting this all wrong !!! > > > > > > And I think this is common requirement for web crawling solutions to > get > > > custom data from page... then are not there any such Nutch plugins > > > available on web ? > > > > > > Thanks, > > > Tony. > > > > > > > > > On Tue, Jun 11, 2013 at 7:35 PM, Markus Jelsma > > > <markus.jel...@openindex.io>wrote: > > > > > > > Hi, > > > > > > > > Yes, you should write a plugin that has a parse filter and indexing > > > > filter. To ease maintenance you would want to have a file per > > host/domain > > > > containing XPath expressions, far easier that switch statements that > > need > > > > to be recompiled. The indexing filter would then index the field > values > > > > extracted by your parse filter. > > > > > > > > Cheers, > > > > Markus > > > > > > > > -----Original message----- > > > > > From:Tony Mullins <tonymullins...@gmail.com> > > > > > Sent: Tue 11-Jun-2013 16:07 > > > > > To: user@nutch.apache.org > > > > > Subject: Data Extraction from 100+ different sites... > > > > > > > > > > Hi, > > > > > > > > > > I have 100+ different sites ( and may be more will be added in near > > > > > future), I have to crawl them and extract my required information > > from > > > > each > > > > > site. So each site would have its own extraction rule ( XPaths). > > > > > > > > > > So far I have seen there is no built-in mechanism in Nutch to > fulfill > > > my > > > > > requirement and I may have to write custom HTMLParserFilter > > extension > > > > and > > > > > IndexFilter plugin. > > > > > > > > > > And I may have to write 100+ switch cases in my plugin to handle > the > > > > > extraction rules of each site.... > > > > > > > > > > Is this the best way to handle my requirement or there is any > better > > > way > > > > to > > > > > handle it ? > > > > > > > > > > Thanks for your support & help. > > > > > > > > > > Tony. > > > > > > > > > > > > > > >
