[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722975#comment-17722975 ] Mingchun Zhao commented on CONNECTORS-1747: --- Hello, If there are no objections to the above patch, would it be okay to commit it in a couple of days? > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Priority: Major > Attachments: JobManager.java.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722789#comment-17722789 ] Mingchun Zhao commented on CONNECTORS-1747: --- Hello, I’ve tried to create a patch for adding the below property to disable logging hopcount to the database as below. "org.apache.manifoldcf.db.postgres.crewler.jobs.store_hopcount" If you do not require hopcount from within, this will disable logging hotcount to the related database tables. This can increase throughput and reduce the rate of growth of the database. defaults to true(logging hopcount to the database). In my testing with the attached patch, I compared the execution time of the same job with the property “store_hopcount” set to true and false. As a result, the throughput doubled and the rate of growth of the database was cut by more than half, while the number of crawled documents remained the same. [^JobManager.java.patch] > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Priority: Major > Attachments: JobManager.java.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (CONNECTORS-1747) Add a property to disable logging hop count to database
[ https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingchun Zhao updated CONNECTORS-1747: -- Attachment: JobManager.java.patch > Add a property to disable logging hop count to database > --- > > Key: CONNECTORS-1747 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1747 > Project: ManifoldCF > Issue Type: Improvement >Reporter: Mingchun Zhao >Priority: Major > Attachments: JobManager.java.patch > > > If we do not require “Hop Filters“ feature, we need to consider to disable > logging records related to hopcount to database like "intrinsiclink" and > "hopcount" tables. This can increase throughput and reduce the rate of growth > of the database. > I will try to create a patch for this. -- This message was sent by Atlassian Jira (v8.20.10#820010)