[
https://issues.apache.org/jira/browse/CONNECTORS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-854:
--------------------------------------
Assignee: Karl Wright
> Enable STALE_CONNECTION_CHECK
> -----------------------------
>
> Key: CONNECTORS-854
> URL: https://issues.apache.org/jira/browse/CONNECTORS-854
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Web connector
> Affects Versions: ManifoldCF 1.4.1
> Reporter: Shinichiro Abe
> Assignee: Karl Wright
> Priority: Minor
> Fix For: ManifoldCF 1.5
>
>
> When crawling some sites( < 1000 docs), sometimes manifoldcf.log shows the
> following "The target server failed to respond" messages. It seems that
> NoHttpResponseException is thrown at ThrottledFetcher.
> {noformat}
> WARN 2014-01-09 12:39:16,701 (Worker thread '10') - Pre-ingest service
> interruption reported for job 1389238470356 connection '1': Timed out waiting
> for response for 'http://www.rondhuit.com/?p=1890': The target server failed
> to respond
> WARN 2014-01-09 12:39:55,509 (Worker thread '7') - Pre-ingest service
> interruption reported for job 1389238470356 connection '1': Timed out waiting
> for response for 'http://www.rondhuit.com/?p=675': The target server failed
> to respond
> {noformat}
> The fetching that page after retry time(15 minutes) passed was running
> successfully.
> I tried to change a httpclient configuration then I confirmed that massage
> was not shown.
> {noformat}
> +++
> connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java
> @@ -463,7 +463,7 @@
> BasicHttpParams params = new BasicHttpParams();
> params.setParameter(ClientPNames.DEFAULT_HOST,fetchHost);
> params.setBooleanParameter(CoreConnectionPNames.TCP_NODELAY,true);
> -
> params.setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK,false);
> +
> params.setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK,true);
>
> params.setBooleanParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS,true);
> {noformat}
> I know two users who are hitting this issue and have resolved it by turning
> on stale connection check.
> The crawling job is done more quickly than the check is false because there
> are not retry fetches.
> May I switch false to true in stale connection check as well as
> SolrConnector's httpclient configuration?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
