As per [0], the prescribed way is to modify ${NUTCH_HOME}/conf/nutch-site.xml
and then compile nutch using 'ant runtime'.[0] : https://wiki.apache.org/nutch/Nutch2Tutorial On Tue, Apr 2, 2013 at 9:38 PM, Yves S. Garret <[email protected]>wrote: > Hello, I'm using 2.x. When I said nutch-site.xml, is it > ${NUTCH_HOME}/conf/nutch-site.xml or > ${NUTCH_HOME}/runtime/local/conf/nutch-site.xml. > > There are two of them and I'm not sure which one is meant to be modified. > > On Tue, Apr 2, 2013 at 11:58 PM, Tejas Patil <[email protected] > >wrote: > > > Hi Yves, > > > > I am able to crawl that url at my end using nutch 1.x trunk. The > > configuration file to be modified is nutch-site.xml which is present > inside > > "conf" directory from where you are running nutch. You need to place the > > property (that I mentioned in my earlier email) in that file. > > > > In case if you are using 2.x, you need to modify the config file as > above, > > re-build nutch using 'ant runtime' command and then run a crawl. > > > > Thanks, > > Tejas > > > > > > > > On Tue, Apr 2, 2013 at 6:58 PM, Yves S. Garret > > <[email protected]>wrote: > > > > > Wait, one more question, which specific nutch-site.xml file should I > > > modify? I know > > > that there are two conf files, so I'd like to confirm that I'm editing > > the > > > correct one. > > > > > > On Tue, Apr 2, 2013 at 8:21 PM, Alvaro Cabrerizo <[email protected]> > > > wrote: > > > > > > > Hello: > > > > > > > > I have had no problem indexing the page > > > > https://plus.google.com/+projectglass (i made a test using this > single > > > > url) > > > > following the instructions pointed by Tejas. I've just changed my > > > > nutch-site.xml removing protocol-http and adding protocol-httpclient > > > > within plugin includes node. > > > > > > > > > > > > Check in your log file (default logs/hadoop.log) if the plugin > > httpclient > > > > is loaded: > > > > > > > > ........... > > > > *2013-04-03 02:03:19,489 INFO plugin.PluginRepository - > Http / > > > > Https Protocol Plug-in (protocol-httpclient)* > > > > *..........* > > > > > > > > Regards > > > > > > > > > > > > On Wed, Apr 3, 2013 at 1:30 AM, Yves S. Garret > > > > <[email protected]>wrote: > > > > > > > > > I don't think anything has changed: > > > > > http://bin.cakephp.org/view/1498883341 > > > > > > > > > > This is my ${NUTCH_HOME}/conf/nutch-site.xml: > > > > > http://bin.cakephp.org/view/883727476 > > > > > > > > > > On Tue, Apr 2, 2013 at 6:51 PM, Tejas Patil < > > [email protected] > > > > > >wrote: > > > > > > > > > > > Looks like the required plugin aint present in the configuration > > file > > > > > > [0].Can you try adding "protocol-httpclient" to plugin.includes > > > > property > > > > > in > > > > > > conf/nutch-site.xml ? Like this > > > > > > > > > > > > <property> > > > > > > <name>plugin.includes</name> > > > > > > > > > > > > > > > > > > > > > > > > > > > <value>protocol-(http|httpclient)|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|indexer-solr|scoring-opic|urlnormalizer-(pass|regex|basic)</value> > > > > > > <description>Regular expression naming plugin directory names > to > > > > > > include. Any plugin not matching this expression is excluded. > > > > > > In any case you need at least include the nutch-extensionpoints > > > > plugin. > > > > > > By > > > > > > default Nutch includes crawling just HTML and plain text via > > HTTP, > > > > > > and basic indexing and search plugins. In order to use HTTPS > > please > > > > > > enable > > > > > > protocol-httpclient, but be aware of possible intermittent > > problems > > > > > with > > > > > > the > > > > > > underlying commons-httpclient library. > > > > > > </description> > > > > > > </property> > > > > > > > > > > > > Let us know if that worked. > > > > > > > > > > > > [0] > > > https://wiki.apache.org/nutch/HttpAuthenticationSchemes#Necessity > > > > > > > > > > > > > > > > > > On Tue, Apr 2, 2013 at 3:27 PM, Yves S. Garret > > > > > > <[email protected]>wrote: > > > > > > > > > > > > > Hi all, just tried crawling this site [ > > > > > > > https://plus.google.com/+projectglass ]: > > > > > > > > > > > > > > This is the output that nutch is showing me: > > > > > > > http://bin.cakephp.org/view/1701305116 > > > > > > > > > > > > > > It seems to be erroring out when it gets to here: > > > > > > > > > > > > > > fetching https://plus.google.com/+projectglass > > > > > > > Unexpected error for https://plus.google.com/+projectglass > > > > > > > org.apache.nutch.protocol.ProtocolNotFound: protocol not found > > for > > > > > > > url=https > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.nutch.protocol.ProtocolFactory.getProtocol(ProtocolFactory.java:83) > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:490) > > > > > > > Fetcher: throughput threshold: -1 > > > > > > > Fetcher: throughput threshold sequence: 5 > > > > > > > > > > > > > > > > > > > > > Why is this happening? > > > > > > > > > > > > > > > > > > > > > > > > > > > >
