[
https://issues.apache.org/jira/browse/CONNECTORS-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835575#comment-13835575
]
Karl Wright commented on CONNECTORS-781:
----------------------------------------
Scheduling on small scale seems to be acceptable; new code base seems to be
reasonably egalitarian among the bins being crawled, but not quite as
egalitarian as I recall from the original code base. Of course, to verify that
I'll need to run the same test on trunk to see if I get roughly the same
numbers, but so far it looks like we don't have any obvious starvation
conditions.
I'm currently tentatively planning to commit this code to trunk on Monday or
Tuesday.
> Fault-Tolerant Setup for ManifoldCF Agent.
> ------------------------------------------
>
> Key: CONNECTORS-781
> URL: https://issues.apache.org/jira/browse/CONNECTORS-781
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Framework agents process, Framework core, Framework
> crawler agent
> Affects Versions: ManifoldCF 1.5
> Reporter: Swami Rajamohan
> Assignee: Karl Wright
> Labels: agents, crawler, fault-tolerance
> Fix For: ManifoldCF 1.5
>
>
> It should be possible to setup ManifoldCF as a Fault-Tolerant infrastructure.
> The Agent component of ManifoldCF should support multiple instances of an
> agent crawling against a single crawl store, to be able to both distribute
> (share) the crawl load as well as to be able to pick up a request that gets
> abruptly terminated due to either partitioning of the instance/failure of the
> instance itself.
> Since there is a proposal to move to a store like Voldemort, it would be nice
> to be able to have a fault tolerant infrastructure.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
