Hi Yves, I am able to crawl that url at my end using nutch 1.x trunk. The configuration file to be modified is nutch-site.xml which is present inside "conf" directory from where you are running nutch. You need to place the property (that I mentioned in my earlier email) in that file.
In case if you are using 2.x, you need to modify the config file as above, re-build nutch using 'ant runtime' command and then run a crawl. Thanks, Tejas On Tue, Apr 2, 2013 at 6:58 PM, Yves S. Garret <[email protected]>wrote: > Wait, one more question, which specific nutch-site.xml file should I > modify? I know > that there are two conf files, so I'd like to confirm that I'm editing the > correct one. > > On Tue, Apr 2, 2013 at 8:21 PM, Alvaro Cabrerizo <[email protected]> > wrote: > > > Hello: > > > > I have had no problem indexing the page > > https://plus.google.com/+projectglass (i made a test using this single > > url) > > following the instructions pointed by Tejas. I've just changed my > > nutch-site.xml removing protocol-http and adding protocol-httpclient > > within plugin includes node. > > > > > > Check in your log file (default logs/hadoop.log) if the plugin httpclient > > is loaded: > > > > ........... > > *2013-04-03 02:03:19,489 INFO plugin.PluginRepository - Http / > > Https Protocol Plug-in (protocol-httpclient)* > > *..........* > > > > Regards > > > > > > On Wed, Apr 3, 2013 at 1:30 AM, Yves S. Garret > > <[email protected]>wrote: > > > > > I don't think anything has changed: > > > http://bin.cakephp.org/view/1498883341 > > > > > > This is my ${NUTCH_HOME}/conf/nutch-site.xml: > > > http://bin.cakephp.org/view/883727476 > > > > > > On Tue, Apr 2, 2013 at 6:51 PM, Tejas Patil <[email protected] > > > >wrote: > > > > > > > Looks like the required plugin aint present in the configuration file > > > > [0].Can you try adding "protocol-httpclient" to plugin.includes > > property > > > in > > > > conf/nutch-site.xml ? Like this > > > > > > > > <property> > > > > <name>plugin.includes</name> > > > > > > > > > > > > > > <value>protocol-(http|httpclient)|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|indexer-solr|scoring-opic|urlnormalizer-(pass|regex|basic)</value> > > > > <description>Regular expression naming plugin directory names to > > > > include. Any plugin not matching this expression is excluded. > > > > In any case you need at least include the nutch-extensionpoints > > plugin. > > > > By > > > > default Nutch includes crawling just HTML and plain text via HTTP, > > > > and basic indexing and search plugins. In order to use HTTPS please > > > > enable > > > > protocol-httpclient, but be aware of possible intermittent problems > > > with > > > > the > > > > underlying commons-httpclient library. > > > > </description> > > > > </property> > > > > > > > > Let us know if that worked. > > > > > > > > [0] > https://wiki.apache.org/nutch/HttpAuthenticationSchemes#Necessity > > > > > > > > > > > > On Tue, Apr 2, 2013 at 3:27 PM, Yves S. Garret > > > > <[email protected]>wrote: > > > > > > > > > Hi all, just tried crawling this site [ > > > > > https://plus.google.com/+projectglass ]: > > > > > > > > > > This is the output that nutch is showing me: > > > > > http://bin.cakephp.org/view/1701305116 > > > > > > > > > > It seems to be erroring out when it gets to here: > > > > > > > > > > fetching https://plus.google.com/+projectglass > > > > > Unexpected error for https://plus.google.com/+projectglass > > > > > org.apache.nutch.protocol.ProtocolNotFound: protocol not found for > > > > > url=https > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.nutch.protocol.ProtocolFactory.getProtocol(ProtocolFactory.java:83) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:490) > > > > > Fetcher: throughput threshold: -1 > > > > > Fetcher: throughput threshold sequence: 5 > > > > > > > > > > > > > > > Why is this happening? > > > > > > > > > > > > > > >
