Yes all the web pages will have different HTML structure/layout and I would have to identify/define a XPath expression for each one of them.
But I am trying to come up with generic output format for these XPath expressions.... so whatever the XPath expression is I want to have result in lets say Field A , Field B , Field C . In some cases some of these fields could be blank as well. So I could map them to my Solr schema properly. In this regard I was hopping to get some help or guideline from your past experiences ... Thanks, Tony On Tue, Jun 11, 2013 at 10:09 PM, AC Nutch <acnu...@gmail.com> wrote: > Hi Tony, > > So if I understand correctly, you have 100+ web pages, each with a totally > different format that you're trying to extract separate/unrelated pieces of > information from. If there's no connection between any of the web pages or > any of the pieces of information that you're trying to extract then it's > pretty much unavoidable to have to provide separate identifiers and cases > for finding each one. Markus' suggestion I believe is to just have a > "dictionary" file with URL as the key and XPath expression for the info > that you want as the value. No matter what crawling/parsing platform you're > using a solution of that sort is pretty much unavoidable with the > assumptions given. > > That being said, is there any common form that the data you're trying to > extract from these pages follows? Is there a regex that could match it or > anything else that might identify it in a common way? > > Alex > > > On Tue, Jun 11, 2013 at 12:59 PM, Tony Mullins <tonymullins...@gmail.com > >wrote: > > > Hi Markus, > > > > I couldn't understand how can I avoid switch cases in your suggested > > idea.... > > > > I would have one plugin which will implement HtmlParseFilter and I would > > have to check the current URL by getting content.getUrl() and this all > will > > be happening in same class so I would have to add swicth cases... I may > > could add xpath expression for each site in separate files but to get > XPath > > expression I would have to decide which file I have to read and for that > I > > would have to add my this code logic in swith case.... > > > > Please correct me if I am getting this all wrong !!! > > > > And I think this is common requirement for web crawling solutions to get > > custom data from page... then are not there any such Nutch plugins > > available on web ? > > > > Thanks, > > Tony. > > > > > > On Tue, Jun 11, 2013 at 7:35 PM, Markus Jelsma > > <markus.jel...@openindex.io>wrote: > > > > > Hi, > > > > > > Yes, you should write a plugin that has a parse filter and indexing > > > filter. To ease maintenance you would want to have a file per > host/domain > > > containing XPath expressions, far easier that switch statements that > need > > > to be recompiled. The indexing filter would then index the field values > > > extracted by your parse filter. > > > > > > Cheers, > > > Markus > > > > > > -----Original message----- > > > > From:Tony Mullins <tonymullins...@gmail.com> > > > > Sent: Tue 11-Jun-2013 16:07 > > > > To: user@nutch.apache.org > > > > Subject: Data Extraction from 100+ different sites... > > > > > > > > Hi, > > > > > > > > I have 100+ different sites ( and may be more will be added in near > > > > future), I have to crawl them and extract my required information > from > > > each > > > > site. So each site would have its own extraction rule ( XPaths). > > > > > > > > So far I have seen there is no built-in mechanism in Nutch to fulfill > > my > > > > requirement and I may have to write custom HTMLParserFilter > extension > > > and > > > > IndexFilter plugin. > > > > > > > > And I may have to write 100+ switch cases in my plugin to handle the > > > > extraction rules of each site.... > > > > > > > > Is this the best way to handle my requirement or there is any better > > way > > > to > > > > handle it ? > > > > > > > > Thanks for your support & help. > > > > > > > > Tony. > > > > > > > > > >
