[
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17722789#comment-17722789
]
Mingchun Zhao commented on CONNECTORS-1747:
-------------------------------------------
Hello,
I’ve tried to create a patch for adding the below property to disable logging
hopcount to the database as below.
"org.apache.manifoldcf.db.postgres.crewler.jobs.store_hopcount"
If you do not require hopcount from within, this will disable logging hotcount
to the related database tables. This can increase throughput and reduce the
rate of growth of the database. defaults to true(logging hopcount to the
database).
In my testing with the attached patch, I compared the execution time of the
same job with the property “store_hopcount” set to true and false. As a result,
the throughput doubled and the rate of growth of the database was cut by more
than half, while the number of crawled documents remained the same.
[^JobManager.java.patch]
> Add a property to disable logging hop count to database
> -------------------------------------------------------
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
> Issue Type: Improvement
> Reporter: Mingchun Zhao
> Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable
> logging records related to hopcount to database like "intrinsiclink" and
> "hopcount" tables. This can increase throughput and reduce the rate of growth
> of the database.
> I will try to create a patch for this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
