[
https://issues.apache.org/jira/browse/CONNECTORS-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736873#comment-16736873
]
Tim Steenbeke commented on CONNECTORS-1562:
-------------------------------------------
Ok I will create a new ticket for both issues.
Testing the connector with the UI, disabling bandwidth and max connections
equal to 20, we were able to crawl all sites.
Now I'm still stuck with the deletion issue, were if a site is removed from the
sitemap, will it be removed from elastic, since it is no longer accessible.
We put the sitemap as seed, the big one, and if 1 or a few URL's get removed,
they should get removed from elastic.
> Documents unreachable due to hopcount are not considered unreachable on
> cleanup pass
> ------------------------------------------------------------------------------------
>
> Key: CONNECTORS-1562
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1562
> Project: ManifoldCF
> Issue Type: Bug
> Components: Elastic Search connector, Web connector
> Affects Versions: ManifoldCF 2.11
> Environment: Manifoldcf 2.11
> Elasticsearch 6.3.2
> Web inputconnector
> elastic outputconnecotr
> Job crawls website input and outputs content to elastic
> Reporter: Tim Steenbeke
> Assignee: Karl Wright
> Priority: Critical
> Labels: starter
> Fix For: ManifoldCF 2.12
>
> Attachments: Screenshot from 2018-12-31 11-17-29.png,
> manifoldcf.log.cleanup, manifoldcf.log.init, manifoldcf.log.reduced
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> My documents aren't removed from ElasticSearch index after rerunning the
> changed seeds
> I update my job to change the seedmap and rerun it or use the schedualer to
> keep it runneng even after updating it.
> After the rerun the unreachable documents don't get deleted.
> It only adds doucments when they can be reached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
