Hi Sebastian,
Here is the index-writers.xml you requested. Thank
Your Sincerely,
Shi Wei
-----Original Message-----
From: Sebastian Nagel <[email protected]>
Sent: Friday, 22 October, 2021 5:46 PM
To: [email protected]
Subject: Re: Cant integrate the kerberos enabled solr cloud with nutch
Hi Shi Wei,
could you also share the index writer configuration (conf/index-writers.xml)?
The default is unauthenticated access to Solr, see the snippet below.
The file httpclient-auth.xml is not relevant for the Solr indexer, it's used if
a crawled web site requires authentication in order to fetch the content via
the plugin protocol-httpclient.
Best,
Sebastian
<writer id="indexer_solr_1"
class="org.apache.nutch.indexwriter.solr.SolrIndexWriter">
<parameters>
<param name="type" value="http"/>
<param name="url" value="http://localhost:8983/solr/nutch"/>
<param name="collection" value=""/>
<param name="weight.field" value=""/>
<param name="commitSize" value="1000"/>
<param name="auth" value="false"/>
<param name="username" value="username"/>
<param name="password" value="password"/>
On 10/22/21 10:10 AM, [email protected] wrote:
> Hi,
>
> We have encountered a problem which can’t integrate the kerberos enabled solr
> cloud with nutch.
>
> When execute "nutch index crawl/crawldb/ -linkdb crawl/linkdb/ $s1
> -filter -normalize" command ,it will fail with "HTTP ERROR 401Problem
> accessing /solr/admin/collections. Reason:Authentication required" but we
> able to curl it with the keytab.
>
> Version of Nutch :1.18
>
> Your Sincerely,
>
> Shi Wei
>
<?xml version="1.0" encoding="UTF-8" ?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<writers xmlns="http://lucene.apache.org/nutch";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://lucene.apache.org/nutch index-writers.xsd">
<writer id="indexer_solr_1" class="org.apache.nutch.indexwriter.solr.SolrIndexWriter">
<parameters>
<param name="type" value="cloud"/>
<param name="url" value="http://utility.quandatics.tech:8983/solr/"/>
<param name="collection" value="nutchapp"/>
<param name="weight.field" value=""/>
<param name="commitSize" value="100"/>
<param name="auth" value="true"/>
<param name="username" value="james"/>
<!-- <param name="username" value="james"/> -->
<!-- <param name="password" value="p@ssw0rd"/> -->
<param name="password" value="p@ssw0rd"/>
</parameters>
<mapping>
<copy>
<!-- <field source="content" dest="search"/> -->
<!-- <field source="title" dest="title,search"/> -->
</copy>
<rename>
<field source="metatag.description" dest="description"/>
<field source="metatag.keywords" dest="keywords"/>
</rename>
<remove>
<field source="segment"/>
</remove>
</mapping>
</writer>
</writers>
