Re: Solr ingestor for Solr 9

[Mingchun Zhao]

Hi Guylaine,

Just FYI, allow me to share my test result with you for your PR:
https://github.com/apache/manifoldcf/pull/161

In my `ant test`, the result was OK, but it took 49 minutes. If an error
occurs due to timeout during your test, it might be better to re-run your
test.

```
% ant clean clean-deps clean-core-deps make-core-deps make-deps build test
... ...
test:

BUILD SUCCESSFUL
Total time: 49 minutes 2 seconds
```

Regards,
Mingchun

2024年1月20日(土) 1:06 Guylaine BASSETTE :

> Hi Karl,
>
> The Solr Ingestor needed a little fix to ingest Solr 9 documents. The
> "httpmime" dependency was missing from the distribution packaging. With
> Solr 9, a bug in Date field filling was discovered and fixed. We are using
> it in our application.
>
> We are currently using these fixes in our application.
>
> I have created a Jira topic for this (
> https://issues.apache.org/jira/projects/CONNECTORS/issues/CONNECTORS-1752)
> and created the corresponding PR:
> https://github.com/apache/manifoldcf/pull/161
>
> Note that I have run "ant test" and all passed except for elasticsearch
> IT-HSQLDB tests. I get back up until 2.25 and they were already showing the
> same failures.
> --
> Best Regards,
> Guylaine
>
> France Labs – Your knowledge, now
> Datafari Enterprise Search – Découvrez la version 5 / Discover our version
> 5
> www.datafari.com
>
> Datafari Enterprise Search – Retrouvez-nous à WAICF Cannes
>  du 8 au 10 février
>
> [image: Logo_temp_waicf_2024] 
>
>
>

Re: MCF Postgres upgrade to 15.4

[Mingchun Zhao]

Hi Guylaine,

Thank you for updating Postgresql. Although I'm not an expert in
PostgreSQL, I'll take a look at the optimization parameters  of PostgreSQL
for ManifoldCF.

Regards,
Mingchun

2023年11月17日(金) 22:52 Karl Wright :

> Generally, Postgresql is pretty stable, but you would want to update the
> JDBC jar for postgresql as well.
> Karl
>
>
> On Fri, Nov 17, 2023 at 6:19 AM Guylaine BASSETTE <
> guylaine.basse...@francelabs.com> wrote:
>
> > Hi all,
> >
> > For what it’s worth, we have upgraded Postgresql to version 15.4 for the
> > MCF 2.26 that is embedded in Datafari, as part of our work towards
> > Datafari 6.0
> >
> > We have run our Datafari tests, and we have not identified any
> > particular issues, so it seems that MCF 2.26 is compatible with it. Note
> > that we have not used any of the embedded testings that are within MCF,
> > so you may want to test that before certifying that it is 100%
> > compatible, but as far as we are concerned it works like a charm.
> >
> > @Mingchun , since you are an expert in Postgresql, would you have some
> > time to look at the optimization parameters of Postgresql for MCF? It’s
> > been quite a while since those have been done for MCF, (probably dating
> > back to Postgresql 9.x), and it is highly possible that new parameters
> > have appeared or have changed.
> >
> >
> > Regards,
> > Guylaine
> >
> > France Labs – Your knowledge, now
> > Datafari Enterprise Search – Découvrez la version 5 / Discover our
> version
> > 5
> > www.datafari.com 
> >
>

Re: [RESULT][VOTE] Release Apache ManifoldCF 2.26, RC1

[Mingchun Zhao]

Hi Guylaine,

Thanks! I just made a few changes based on your research.

Kind regards,
Mingchun


2023年11月8日(水) 16:24 Guylaine BASSETTE :

> Hi,
>
> Thank you Mingchun for your last fix on MockSolrService.
>
> Great to see ManifoldCF 2.26 has been released ! Thanks all !
>
> Le 01/11/2023 à 17:48, Karl Wright a écrit :
> > Three +1's, >72 hours.  Vote passes!
> >
> > Karl
> >
> >
> > On Wed, Nov 1, 2023 at 12:47 PM Karl Wright  wrote:
> >
> >> +1 from me.
> >> Karl
> >>
> >>
> >> On Sun, Oct 29, 2023 at 4:47 AM Furkan KAMACI
> >> wrote:
> >>
> >>> +1
> >>>
> >>> On Sun, Oct 29, 2023 at 2:00 AM Mingchun Zhao<
> mingchun.zha...@gmail.com>
> >>> wrote:
> >>>
> >>>> +1
> >>>>
> >>>> Built and tested from tag release-2.26-RC1
> >>>> <http://svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC1>
> with
> >>>> Ant.
> >>>>
> >>>> OS name: macOS 14.0
> >>>> Apache Ant(TM) version 1.10.0
> >>>> Java version: openjdk version "11.0.11"
> >>>> locale: en_US.UTF-8
> >>>>
> >>>> Thanks for doing the release!
> >>>>
> >>>> Regards,
> >>>> Mingchun
> >>>>
> >>>> 2023年10月29日(日) 1:12 Karl Wright:
> >>>>
> >>>>> Please vote on whether to release Apache ManifoldCF 2.26, RC1.  The
> >>>> release
> >>>>> candidate can be found here:
> >>>>>
> >>>>>
> >>>
> https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.26
> >>>>> There is also a release tag at:
> >>>>>
> >>>>> https//svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC1
> >>>>>
> >>>>> Thanks to Mingchun Zhao and Guylaine Bassette for making this release
> >>>>> possible!
> >>>>>
> >>>>> Karl
> >>>>>
> --
> Cordialement,
> Guylaine
>
> France Labs – Your knowledge, now
> Datafari Enterprise Search – Découvrez la version 5 / Discover our version
> 5
> www.datafari.com <http://www.datafari.com>

Re: 2.26 release documentation did not build

[Mingchun Zhao]

> Reason: the switch to java 11 meant that Forrest did not compile.

Thanks. I'll also try updating the Forrest version a bit.

2023年11月2日(木) 8:15 Karl Wright :

> Reason: the switch to java 11 meant that Forrest did not compile.
> Apparently the Java 11 compiler will no longer handle the earlier source
> versions specified in the Forrest build.
>
> I'm still going to update the website, but we're going to also now need to
> invest in perhaps updating which Forrest version we take.  This is not
> trivial because I had to make customizations to make the PDF generator used
> the fonts I downloaded for this, and IIRC Forrest later changed in a way
> which broke my customizations.
>
> Karl
>

Re: [VOTE] Release Apache ManifoldCF 2.26, RC1

[Mingchun Zhao]

+1

Built and tested from tag release-2.26-RC1
<http://svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC1> with Ant.

OS name: macOS 14.0
Apache Ant(TM) version 1.10.0
Java version: openjdk version "11.0.11"
locale: en_US.UTF-8

Thanks for doing the release!

Regards,
Mingchun

2023年10月29日(日) 1:12 Karl Wright :

> Please vote on whether to release Apache ManifoldCF 2.26, RC1.  The release
> candidate can be found here:
>
> https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.26
>
> There is also a release tag at:
>
> https//svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC1
>
> Thanks to Mingchun Zhao and Guylaine Bassette for making this release
> possible!
>
> Karl
>

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Hi There,
In order to resolve junit test failure with Solr 9.x Output connector, I
modified MockSolrService to support HTTP2C.
I prepared a PR for this: https://github.com/apache/manifoldcf/pull/157
I confirmed that all test cases of ant test passed.
```
test:
BUILD SUCCESSFUL
Total time: 51 minutes 37 seconds
```

Regards,
Mingchun

2023年10月28日(土) 1:50 Guylaine BASSETTE :

> Just a precision about the Solr NavigationHSQLDBUI test was talking
> about, my problem is about the test environment I failed to configure
> properly in the time I got to work on it. So I don't know if it could
> succeed or not.
>
> Guylaine
> Le 27/10/2023 à 17:34, Guylaine BASSETTE a écrit :
>
> Hi Karl,
>
> The errors shown by Mingchun were puzzling me, so I did some more testing:
> actually, if we set up a real solr 9 server, and modify the Solr test to
> point to it rather than to the mockupsolr, then everything runs fine, the
> tests ends successfully and the data are actually uploaded into solr 9. My
> conclusion is that the error is therefore not at all related to conflicting
> libraries, but probably to the mockupsolrservice that is now somehow
> outdated in the way that it interacts with the jetty client.
>
>
> Joint to this mail, the logs in Solr server and the test results (I have
> added extra logs on job status when I tested)
>
> Here are the modification I have made:
> Remove Mock server launch in SolrCrawlHSQLDBIT class:
>
> public class SolrCrawlHSQLDBIT extends BaseITHSQLDB
> {
>
>   protected SolrTester tester;
>   protected MockSolrService solrService = null;
>
>   public SolrCrawlHSQLDBIT()
>   {
> tester = new SolrTester(mcfInstance);
>   }
>
>   @Test
>   public void simpleCrawl()
> throws Exception
>   {
> tester.executeTest();
>   }
> }
>
>
> In SolrTester:
>
> configParams.setParameter(SolrConfig.PARAM_SERVER,"IP_SOLR_SERVER");
> // To be replaced with the Solr server IP
> configParams.setParameter(SolrConfig.PARAM_PORT,"8983");
>
> configParams.setParameter(SolrConfig.PARAM_WEBAPPNAME,"solr/techproducts");
> configParams.setParameter(SolrConfig.PARAM_COLLECTION, "solr");
>
> I was not able to pass the NavigationHSQLDBUI test.
>
> Sorry, I won't be available next week to go further. I hope this will help
> !
>
> Guylaine
> Le 27/10/2023 à 13:40, Karl Wright a écrit :
>
> That is indeed reassuring.  It probably means that the test needs some
> changes, is all.  But we cannot be sure the newer Zookeeper wouldn't mess
> things up without having the test be successful at least with the older
> zookeeper.
>
> Karl
>
>
> On Fri, Oct 27, 2023 at 7:32 AM Guylaine BASSETTE 
>  wrote:
>
>
> Sorry for not executing all of the tests: I'm quite new to the MCF
> project, and I focused only on the core-framework tests as it was the
> key aspect highlighted in the related jira issue. I didn't know I had to
> run other tests such as the IT ones. It is to be noted that we tested
> the MCF with Solr 9 embedded in our Datafari, and after quite some
> indexing, we have seen no problems at all, which sounds quite reassuring
>
> Le 27/10/2023 à 12:49, Karl Wright a écrit :
>
> Okay, well I wouldnt have approved the upgrade had I known that the tests
> didn't pass!  So we need to understand the problem as soon as possible.
>
> Karl
>
>
> On Fri, Oct 27, 2023 at 2:50 AM Guylaine BASSETTE 
>  wrote:
>
>
> Hello all,
>
> Sadly, the error remains the same even with the previous Zookeeper
> version (3.8.0). Actually, I'm not able to pass the test since we have
> passed to Solr 9.
>
> Le 27/10/2023 à 01:51, Karl Wright a écrit :
>
> It is possible that Solr needs the older version of Zookeeper. If you
> swap out the current one and replace it with the one the version of
> SolrJ we use references, does the test pass then? If it does, we're
> going to have to figure out how to address the fact that we have two
> connectors that each depend on a different version of zookeeper. But
> first please let me know if it works. I'll suggest a way of
> reconciling these once I know.
>
> --
> Cordialement,
> Guylaine
>
> France Labs – Your knowledge, now
> Datafari Enterprise Search – Découvrez la version 5 / Discover our
>
> version
>
> 5www.datafari.com   
>
> --
> Cordialement,
> Guylaine
>
> France Labs – Your knowledge, now
> Datafari Enterprise Search – Découvrez la version 5 / Discover our version
> 5www.datafari.com  
>
> --
> Cordialement,
> Guylaine
>
> France Labs – Your knowledge, now
> Datafari Enterprise Search – Découvrez la version 5 / Discover our version
> 5
> www.datafari.com
>
> --
> Cordialement,
> Guylaine
>
> France Labs – Your knowledge, now
> Datafari Enterprise Search – Découvrez la version 5 / Discover our version
> 5
> www.datafari.com
>

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

I ran the ant test using Guylaine's patch and the Kafka IT test passed!
However, the Solr IT test failure still occurs.
```
run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
[junit] Configuration file successfully read
[junit] [main] INFO org.eclipse.jetty.util.log - Logging initialized
@6974ms to org.eclipse.jetty.util.log.Slf4jLog
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0 Scavenging
every 66ms
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@4565a70a{ManifoldCF Crawler
Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1frgn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-3250496980318085250/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/guylaine/manifoldcf/dist/web/war/mcf-crawler-ui.war}
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@a0a9fa5{ManifoldCF Authorities API
Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1frgn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-6182863812882423778/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/guylaine/manifoldcf/dist/web/war/mcf-authority-service.war}
[junit] Creating mock service
[junit] Mock service created
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@15723761{ManifoldCF General API
Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1frgn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-14231646742487500933/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/guylaine/manifoldcf/dist/web/war/mcf-api-service.war}
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@6f63c44f{HTTP/1.1, (http/1.1)}{0.0.0.0:8346}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started @9062ms
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0 Scavenging
every 66ms
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.s.ServletContextHandler@6b2e0f78{/solr,null,AVAILABLE}
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@3ec2ecea{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started @9071ms
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Stopped ServerConnector@3ec2ecea{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.session - node0 Stopped
scavenging
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Stopped o.e.j.s.ServletContextHandler@6b2e0f78{/solr,null,STOPPED}
```

2023年10月27日(金) 0:35 Guylaine BASSETTE :

> Hi Mingchun,
>
> If you want, you can access my repository directly:
> https://github.com/guylaine-bassette/manifoldcf
>
> I have made a Fork of Github ManifoldCF. Use the branch:
> Fix-kafka-IT-tests. It is up to date with the last trunk commit.
>
> Le 26/10/2023 à 17:13, Mingchun Zhao a écrit :
> > Sorry, there was a typo: track -> trunk
> >
> > 2023年10月27日(金) 0:12 Mingchun Zhao:
> >
> >> Hi Guylaine, Thanks!
> >> Was your patch based on the latest track? Unfortunately, I was unable to
> >> apply your patch in my environment.
> >>
> >> 2023年10月26日(木) 20:44 Guylaine BASSETTE >:
> >>
> >>> Hello Mingchun,
> >>>
> >>> As mentioned in my previous e-mail to Karl, my patch contained other
> >>> files to be updated. I don't know why they don't have been taken into
> >>> account, but would you be interested in trying my patch? I have fixed
> the
> >>> spots I had missed... (joint with this mail)
> >>>
> >>> With these modifications I went through all tests of kafka.
> >>> Le 26/10/2023 à 03:47, Mingchun Zhao a écrit :
> >>>
> >>> Hi there, Allow me to share my `ant test` result us

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Sorry, there was a typo: track -> trunk

2023年10月27日(金) 0:12 Mingchun Zhao :

> Hi Guylaine, Thanks!
> Was your patch based on the latest track? Unfortunately, I was unable to
> apply your patch in my environment.
>
> 2023年10月26日(木) 20:44 Guylaine BASSETTE :
>
>> Hello Mingchun,
>>
>> As mentioned in my previous e-mail to Karl, my patch contained other
>> files to be updated. I don't know why they don't have been taken into
>> account, but would you be interested in trying my patch? I have fixed the
>> spots I had missed... (joint with this mail)
>>
>> With these modifications I went through all tests of kafka.
>> Le 26/10/2023 à 03:47, Mingchun Zhao a écrit :
>>
>> Hi there, Allow me to share my `ant test` result using the latest Github
>> trunk.
>> I got a lot of `Broker may not be available` warnings in the Kafka IT test
>> and it eventually failed.
>>
>> ```
>> ...
>> [junit] [kafka-producer-network-thread | producer-3] INFO
>> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3]
>> Node 0 disconnected.
>> [junit] [kafka-producer-network-thread | producer-3] WARN
>> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3]
>> Connection to node 0 (/192.168.10.103:9092) could not be established.
>> Broker may not be available.
>> [junit] [kafka-producer-network-thread | producer-1] INFO
>> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1]
>> Node 0 disconnected.
>> [junit] [kafka-producer-network-thread | producer-1] WARN
>> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1]
>> Connection to node 0 (/192.168.10.103:9092) could not be established.
>> Broker may not be available.
>> [junit] [kafka-producer-network-thread | producer-2] INFO
>> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2]
>> Node 0 disconnected.
>> [junit] [kafka-producer-network-thread | producer-2] WARN
>> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2]
>> Connection to node 0 (/192.168.10.103:9092) could not be established.
>> Broker may not be available.
>> [junit] [kafka-producer-network-thread | producer-3] INFO
>> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3]
>> Node 0 disconnected.
>> [junit] [kafka-producer-network-thread | producer-3] WARN
>> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3]
>> Connection to node 0 (/192.168.10.103:9092) could not be established.
>> Broker may not be available.
>> [junit] -  ---
>> [junit] Testcase:
>> sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
>> Caused an ERROR
>> [junit] ManifoldCF did not terminate in the allotted time of 12
>> milliseconds
>> [junit] org.apache.manifoldcf.core.interfaces.ManifoldCFException:
>> ManifoldCF did not terminate in the allotted time of 12 milliseconds
>> [junit] at
>> org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT.waitJobInactive(APISanityHSQLDBIT.java:289)
>> [junit] at
>> org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT.sanityCheck(APISanityHSQLDBIT.java:177)
>> [junit] at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> [junit] at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> [junit] at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [junit]
>> [junit]
>>
>> BUILD FAILED
>> /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following
>> error occurred while executing this line:
>> /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following
>> error occurred while executing this line:
>> /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1104:
>> Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed
>>
>> Total time: 38 minutes 13 seconds
>> ```
>>
>>
>> 2023年10月26日(木) 5:42 Karl Wright  :
>>
>>
>> I see you have committed this.  You missed a few spots; the extra jar was
>> mentioned in multiple places.  I committed another fix to correct that.
>>
>> Karl
>>
>>
>> On Wed, Oct 25, 2023 at 10:46 AM Guylaine BASSETTE 
>>  wrote:
>>
>>
>> Hello Karl,
>>
>> Thank you very much for this update! I have tested 

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Hi Guylaine, Thanks!
Was your patch based on the latest track? Unfortunately, I was unable to
apply your patch in my environment.

2023年10月26日(木) 20:44 Guylaine BASSETTE :

> Hello Mingchun,
>
> As mentioned in my previous e-mail to Karl, my patch contained other files
> to be updated. I don't know why they don't have been taken into account,
> but would you be interested in trying my patch? I have fixed the spots I
> had missed... (joint with this mail)
>
> With these modifications I went through all tests of kafka.
> Le 26/10/2023 à 03:47, Mingchun Zhao a écrit :
>
> Hi there, Allow me to share my `ant test` result using the latest Github
> trunk.
> I got a lot of `Broker may not be available` warnings in the Kafka IT test
> and it eventually failed.
>
> ```
> ...
> [junit] [kafka-producer-network-thread | producer-3] INFO
> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3]
> Node 0 disconnected.
> [junit] [kafka-producer-network-thread | producer-3] WARN
> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3]
> Connection to node 0 (/192.168.10.103:9092) could not be established.
> Broker may not be available.
> [junit] [kafka-producer-network-thread | producer-1] INFO
> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1]
> Node 0 disconnected.
> [junit] [kafka-producer-network-thread | producer-1] WARN
> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1]
> Connection to node 0 (/192.168.10.103:9092) could not be established.
> Broker may not be available.
> [junit] [kafka-producer-network-thread | producer-2] INFO
> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2]
> Node 0 disconnected.
> [junit] [kafka-producer-network-thread | producer-2] WARN
> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2]
> Connection to node 0 (/192.168.10.103:9092) could not be established.
> Broker may not be available.
> [junit] [kafka-producer-network-thread | producer-3] INFO
> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3]
> Node 0 disconnected.
> [junit] [kafka-producer-network-thread | producer-3] WARN
> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3]
> Connection to node 0 (/192.168.10.103:9092) could not be established.
> Broker may not be available.
> [junit] -  ---
> [junit] Testcase:
> sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
> Caused an ERROR
> [junit] ManifoldCF did not terminate in the allotted time of 12
> milliseconds
> [junit] org.apache.manifoldcf.core.interfaces.ManifoldCFException:
> ManifoldCF did not terminate in the allotted time of 12 milliseconds
> [junit] at
> org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT.waitJobInactive(APISanityHSQLDBIT.java:289)
> [junit] at
> org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT.sanityCheck(APISanityHSQLDBIT.java:177)
> [junit] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> [junit] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [junit] at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit]
> [junit]
>
> BUILD FAILED
> /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following
> error occurred while executing this line:
> /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following
> error occurred while executing this line:
> /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1104:
> Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed
>
> Total time: 38 minutes 13 seconds
> ```
>
>
> 2023年10月26日(木) 5:42 Karl Wright  :
>
>
> I see you have committed this.  You missed a few spots; the extra jar was
> mentioned in multiple places.  I committed another fix to correct that.
>
> Karl
>
>
> On Wed, Oct 25, 2023 at 10:46 AM Guylaine BASSETTE 
>  wrote:
>
>
> Hello Karl,
>
> Thank you very much for this update! I have tested your suggestions and
> kafka ITs tests ended successfully. :-)
>
> Here you can find the patch.
>
> My only doubt is this warning I had in some parts of the kafka test:
>
> ```
>
> [junit] [Controller-0-to-broker-0-send-thread] INFO
> org.apache.kafka.clients.NetworkClient - [Controller id=0,
> targetBrokerId=0] Node 0 disconnected.
> [junit] [Controller-0-to-broker-0-send-thread] WARN
> org.apache.kafka.clients.NetworkClient - [Contro

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

; > Regarding Kafka tests, a light search brings me to the
> > "spark-streaming-kafka" dependency that might be missing...
> >
> > At the mean time, I continue my effort on Solr connector tests.
> >
> >
> > For any use, here is the errors I get:
> >
> > ```
> >
> >[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
> > Stopped o.e.j.w.WebAppContext@7c28c1{ManifoldCF General API
> >
> >
> Webapp,/mcf-api-service,null,STOPPED}{/home/guylaine/IdeaProjects/mon-manifoldcf/dist/web/war/mcf-api-service.war}
> >  [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler
> > - Stopped o.e.j.w.WebAppContext@588ffeb{ManifoldCF Authorities API
> >
> >
> Webapp,/mcf-authority-service,null,STOPPED}{/home/guylaine/IdeaProjects/mon-manifoldcf/dist/web/war/mcf-authority-service.war}
> >  [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler
> > - Stopped o.e.j.w.WebAppContext@71a3a190{ManifoldCF Crawler
> >
> >
> Interface,/mcf-crawler-ui,null,STOPPED}{/home/guylaine/IdeaProjects/mon-manifoldcf/dist/web/war/mcf-crawler-ui.war}
> >  [junit] -  ---
> >  [junit] Testcase:
> > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
> > Caused an ERROR
> >  [junit] 'scala.collection.immutable.ArraySeq
> > scala.runtime.ScalaRunTime$.wrapRefArray(java.lang.Object[])'
> >  [junit] java.lang.NoSuchMethodError:
> > 'scala.collection.immutable.ArraySeq
> > scala.runtime.ScalaRunTime$.wrapRefArray(java.lang.Object[])'
> >  [junit] at
> > kafka.server.KafkaConfig$.(KafkaConfig.scala:338)
> >  [junit] at
> > kafka.server.KafkaConfig.(KafkaConfig.scala:1603)
> >  [junit] at
> >
> >
> org.apache.manifoldcf.agents.output.kafka.KafkaLocal.(KafkaLocal.java:31)
> >  [junit] at
> >
> >
> org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.setupKafka(BaseITHSQLDB.java:86)
> >  [junit] at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >  [junit] at
> >
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >  [junit] at
> >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >  [junit]
> >  [junit]
> >  [junit] Testcase:
> > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
> > Caused an ERROR
> >  [junit] null
> >  [junit] java.lang.NullPointerException
> >  [junit] at
> >
> >
> org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.cleanUpKafka(BaseITHSQLDB.java:92)
> >  [junit] at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >  [junit] at
> >
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >  [junit] at
> >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > ```
> >
> > Le 24/10/2023 à 12:30, Karl Wright a écrit :
> >
> > I missed a place - connector-build.xml.  Updated now.
> >
> > Now we don't get a link exception, but neither does the kafka test work.
> > It seems to be unable to start zookeeper even though all the
> >
> > dependencies
> >
> > are now there.  Will need to look at this after work.
> >
> > Karl
> >
> >
> > On Mon, Oct 23, 2023 at 11:32 PM Mingchun Zhao<
> >
> > mingchun.zha...@gmail.com>
> >
> > wrote:
> >
> >
> > Thanks. I've tried `ant test` with the latest trunk. As a result, the
> >
> > kafka
> >
> > test failed as below.
> > ```
> >  [junit] Testcase:
> >
> >
> > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
> >
> > Caused an ERROR
> >  [junit] io/netty/handler/ssl/SslContext
> >  [junit] java.lang.NoClassDefFoundError:
> >
> > io/netty/handler/ssl/SslContext
> >
> >  [junit] at
> >
> >
> >
> >
> org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106)
> >
> >  [junit] at
> >
> >
> >
> >
> org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Thanks. I've tried `ant test` with the latest trunk. As a result, the kafka
test failed as below.
```
[junit] Testcase:
sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
Caused an ERROR
[junit] io/netty/handler/ssl/SslContext
[junit] java.lang.NoClassDefFoundError: io/netty/handler/ssl/SslContext
[junit] at
org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96)
[junit] at
org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92)
[junit] at
org.apache.zookeeper.common.ZKConfig.(ZKConfig.java:61)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.(ZKClientConfig.java:69)
[junit] at kafka.server.KafkaConfig.(KafkaConfig.scala:1620)
[junit] at kafka.server.KafkaConfig.(KafkaConfig.scala:1603)
[junit] at
org.apache.manifoldcf.agents.output.kafka.KafkaLocal.(KafkaLocal.java:31)
[junit] at
org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.setupKafka(BaseITHSQLDB.java:86)
[junit] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] Caused by: java.lang.ClassNotFoundException:
io.netty.handler.ssl.SslContext
[junit] at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
[junit] at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
[junit] at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
[junit]
[junit]
[junit] Testcase:
sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
Caused an ERROR
[junit] null
[junit] java.lang.NullPointerException
[junit] at
org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.cleanUpKafka(BaseITHSQLDB.java:92)
[junit] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit]
[junit]

BUILD FAILED
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102:
Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed
```

2023年10月24日(火) 11:00 Karl Wright :

> Okay, I updated zookeeper properly in build.xml and framework/build.xml,
> with the two new dependencies, and the zookeeper tests pass.  I haven't
> tried the kafka or solr tests yet.
>
> Karl
>
>
> On Mon, Oct 23, 2023 at 9:29 PM Karl Wright  wrote:
>
> > Unless I know what kafka is using zookeeper for, this would seem risky to
> > me.  Zookeeper is meant to coordinate processes; it may not work for one
> > process to be using different versions of zookeeper than the others.
> >
> > It looks like the original change to kafka you reverted had the proper
> > dependencies but they absolutely needed to be included in the right
> > classpaths and they weren't - they were only included in the kafka tests.
> > I will look at this perhaps at the latest this weekend, but I won't
> commit
> > this patch.
> >
> >
> > Karl
> >
> >
> > On Mon, Oct 23, 2023 at 5:14 PM Mingchun Zhao  >
> > wrote:
> >
> >> I reverted zookeeper version to 3.8.0 to avoid linkage error on the
> >> multiThreadZooKeeperLockTest:
> >> [junit] Caused by: java.lang.ClassNotFoundException:
> >> io.netty.handler.ssl.SslContext
> >>
> >> I've prepared a PR here:
> >> https://github.com/apache/manifoldcf/pull/156
> >>
> >> Just a heads up, `ant test` still hangs on the Solr Output connector
> test:
> >> ```
> >> run-IT-HSQLDB:
> >> [junit] Testsuite:
> >> org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
> >> [junit] Configuration file successfully read
> >> [junit] [main] INFO org.eclipse.jetty.util.log - Logging initialized
> >> @7027ms to org.eclipse.jetty.util.log.Slf4jLog
> >> [junit] [main] INFO org.eclipse.jetty.server.Server -
> >> jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
> >

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

I reverted zookeeper version to 3.8.0 to avoid linkage error on the
multiThreadZooKeeperLockTest:
[junit] Caused by: java.lang.ClassNotFoundException:
io.netty.handler.ssl.SslContext

I've prepared a PR here:
https://github.com/apache/manifoldcf/pull/156

Just a heads up, `ant test` still hangs on the Solr Output connector test:
```
run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
[junit] Configuration file successfully read
[junit] [main] INFO org.eclipse.jetty.util.log - Logging initialized
@7027ms to org.eclipse.jetty.util.log.Slf4jLog
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0 Scavenging
every 66ms
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@1517f633{ManifoldCF Crawler
Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1frgn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-7554899724821045499/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-crawler-ui.war}
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@4fe01803{ManifoldCF Authorities API
Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1frgn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-7701836901953162228/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-authority-service.war}
[junit] Creating mock service
[junit] Mock service created
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@13d186db{ManifoldCF General API
Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1frgn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-2609388202403972652/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-api-service.war}
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@3bd55d8{HTTP/1.1, (http/1.1)}{0.0.0.0:8346}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started @9054ms
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0 Scavenging
every 60ms
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.s.ServletContextHandler@6f4ade6e{/solr,null,AVAILABLE}
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@30e6a763{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started @9064ms
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Stopped ServerConnector@30e6a763{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.session - node0 Stopped
scavenging
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Stopped o.e.j.s.ServletContextHandler@6f4ade6e{/solr,null,STOPPED}
```

2023年10月24日(火) 1:40 Karl Wright :

> The dependencies would be in the zookeeper pom.  Maven would follow them
> automatically which is why it is insufficient to assume that if maven works
> so will ant.
>
> You can use mvn dependency:tree to find what Maven is actually pulling in.
>
> Karl
>
>
> On Mon, Oct 23, 2023 at 11:04 AM Guylaine BASSETTE <
> guylaine.basse...@francelabs.com> wrote:
>
> > I launch these tests with a Maven build and everything is OK. Yet no
> > netty dependencies are requiered... I don't understand were this
> > SslContext is called...
> >
> > Le 23/10/2023 à 16:25, Karl Wright a écrit :
> > > Yes, that is indicating that zookeeper is looking for a specific netty
> > > class that it isn't finding.  That is why I think there is now a
> > zookeeper
> > > dependency we aren't including in the classpaths that include
> zookeeper.
> > >
> > > Karl
> > >
> > >
> > > On Mon, Oct 23, 2023 at 10:23 AM Mingchun Zhao<
> mingchun.zha...@gmail.com
> > >
> > > wrote:
> > >
> > >> Karl, Thanks!
> > >> I 

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Karl, Thanks!
I think I reproduced that multiThreadZooKeeperLockTest error when I
running `ant test`, will look into this.
```
[junit] -  ---
[junit] Testcase:
multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks):
Caused an ERROR
[junit] io/netty/handler/ssl/SslContext
[junit] java.lang.NoClassDefFoundError: io/netty/handler/ssl/SslContext
[junit] at
org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96)
[junit] at
org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92)
[junit] at
org.apache.zookeeper.common.ZKConfig.(ZKConfig.java:61)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.(ZKClientConfig.java:69)
[junit] at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:643)
[junit] at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:567)
[junit] at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:734)
[junit] at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:448)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.(ZooKeeperConnection.java:66)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperConnectionPool.grab(ZooKeeperConnectionPool.java:48)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.obtainGlobalReadLock(ZooKeeperLockObject.java:190)
[junit] at
org.apache.manifoldcf.core.lockmanager.LockObject.enterReadLock(LockObject.java:310)
[junit] at
org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:271)
[junit] at
org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks.enterReadLock(TestZooKeeperLocks.java:125)
[junit] at
org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks$ReaderThread.run(TestZooKeeperLocks.java:204)
[junit] Caused by: java.lang.ClassNotFoundException:
io.netty.handler.ssl.SslContext
[junit] at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
[junit] at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
[junit] at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
[junit]
[junit]

BUILD FAILED
```

2023年10月23日(月) 23:02 Karl Wright :

> This is all you need to do:
>
> ant clean-core-deps
> ant make-core-deps
> ant clean
> ant test
>
> Karl
>
>
> On Mon, Oct 23, 2023 at 9:55 AM Mingchun Zhao 
> wrote:
>
> > Hi Guylaine, Thanks!
> >
> > > Thanks for all your sharing, it's very helpful! I'll continue...
> >
> > I'll look into it some more too. If I have any other information I'll
> share
> > it with you.
> >
> > 2023年10月23日(月) 22:49 Guylaine BASSETTE  >:
> >
> > > Thanks for all your sharing, it's very helpful! I'll continue...
> > >
> > > Sorry, French and English mixed up!
> > >
> > > Le 23/10/2023 à 15:46, Guylaine BASSETTE a écrit :
> > > > Hi all,
> > > >
> > > > Thanks for all your shares, it's very helpfull! Merci pour tous vos
> > > > partages, c'est très utile ! Je poursuis...
> > > >
> > > > Le 23/10/2023 à 15:31, Karl Wright a écrit :
> > > >> I can't give advice on the test; this is something FranceLabs should
> > > >> look
> > > >> at.
> > > >> However, nothing of what you are doing will affect the zookeeper
> > > >> tests in
> > > >> framework.  That's a totally different issue.
> > > >>
> > > >> Karl
> > > >>
> > > >>
> > > >> On Mon, Oct 23, 2023 at 9:20 AM Mingchun Zhao<
> > mingchun.zha...@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> Hi Karl and Guylaine,
> > > >>>
> > > >>>> I hope and think it's just a problem specific to the test. Missing
> > > >>> updates or incompatible dependencies...
> > > >>>
> > > >>> Allow me to share with you what I'm working on.  I've tried to
> > support
> > > >>> http2C within the Solr output connector junit test, but got another
> > > >>> unhandled solr exception when I ran `ant run-IT-HSQLDB`.
> > > >>>
> > > >>> - source code change
> > > >>> ```
> > > >>> diff --git
> > > >>>
> > &

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Hi Guylaine, Thanks!

> Thanks for all your sharing, it's very helpful! I'll continue...

I'll look into it some more too. If I have any other information I'll share
it with you.

2023年10月23日(月) 22:49 Guylaine BASSETTE :

> Thanks for all your sharing, it's very helpful! I'll continue...
>
> Sorry, French and English mixed up!
>
> Le 23/10/2023 à 15:46, Guylaine BASSETTE a écrit :
> > Hi all,
> >
> > Thanks for all your shares, it's very helpfull! Merci pour tous vos
> > partages, c'est très utile ! Je poursuis...
> >
> > Le 23/10/2023 à 15:31, Karl Wright a écrit :
> >> I can't give advice on the test; this is something FranceLabs should
> >> look
> >> at.
> >> However, nothing of what you are doing will affect the zookeeper
> >> tests in
> >> framework.  That's a totally different issue.
> >>
> >> Karl
> >>
> >>
> >> On Mon, Oct 23, 2023 at 9:20 AM Mingchun Zhao >
> >> wrote:
> >>
> >>> Hi Karl and Guylaine,
> >>>
> >>>> I hope and think it's just a problem specific to the test. Missing
> >>> updates or incompatible dependencies...
> >>>
> >>> Allow me to share with you what I'm working on.  I've tried to support
> >>> http2C within the Solr output connector junit test, but got another
> >>> unhandled solr exception when I ran `ant run-IT-HSQLDB`.
> >>>
> >>> - source code change
> >>> ```
> >>> diff --git
> >>>
> >>>
> a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests
>
> >>>
> >>> /MockSolrService.java
> >>>
> >>>
> b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output
>
> >>>
> >>> /solr/tests/MockSolrService.java
> >>> index 237ade09c..3fb558f52 100644
> >>> ---
> >>>
> >>>
> a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo
>
> >>>
> >>> lrService.java
> >>> +++
> >>>
> >>>
> b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo
>
> >>>
> >>> lrService.java
> >>> @@ -18,7 +18,10 @@
> >>>   */
> >>>   package org.apache.manifoldcf.agents.output.solr.tests;
> >>>
> >>> +import org.eclipse.jetty.http2.server.HTTP2CServerConnectionFactory;
> >>>   import org.eclipse.jetty.servlet.ServletHolder;
> >>> +import org.eclipse.jetty.server.HttpConfiguration;
> >>> +import org.eclipse.jetty.server.HttpConnectionFactory;
> >>>   import org.eclipse.jetty.server.Server;
> >>>   import org.eclipse.jetty.server.ServerConnector;
> >>>   import org.eclipse.jetty.servlet.ServletContextHandler;
> >>> @@ -40,7 +43,10 @@ public class MockSolrService
> >>> public MockSolrService()
> >>> {
> >>>   server = new Server(new QueuedThreadPool(35));
> >>> -ServerConnector connector = new ServerConnector(server);
> >>> +HttpConfiguration config = new HttpConfiguration();
> >>> +HttpConnectionFactory http1 = new HttpConnectionFactory(config);
> >>> +HTTP2CServerConnectionFactory http2c = new
> >>> HTTP2CServerConnectionFactory(config);
> >>> +ServerConnector connector = new ServerConnector(server, http1,
> >>> http2c);
> >>>   connector.setPort(8188);
> >>>   server.addConnector(connector);
> >>>   servlet = new SolrServlet();
> >>> @@ -111,6 +117,7 @@ public class MockSolrService
> >>> res.getWriter().printf(Locale.ROOT, "\n");
> >>> res.getWriter().printf(Locale.ROOT, "\n");
> >>> res.getWriter().flush();
> >>>   }
> >>> ```
> >>>
> >>> - run Junit test
> >>> ```
> >>> ~manifoldcf/connectors/solr% ant run-IT-HSQLDB
> >>> ```
> >>>
> >>> - confirm test-HSQLDB-output/manifoldcf.log
> >>> ```
> >>> ERROR 2023-10-23T22:10:29,902 (Worker thread '24') - Exception tossed:
> >>> Unhandled Solr exception during indexinghttp://test70.txt (200): Error
> >>> from server athttp://localhost:8188/solr: Expected mime type
> >>> application/octet-stream but got application/xml. 
> >>>
> >>> 
>

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Hi Karl and Guylaine,

> I hope and think it's just a problem specific to the test. Missing
updates or incompatible dependencies...

Allow me to share with you what I'm working on.  I've tried to support
http2C within the Solr output connector junit test, but got another
unhandled solr exception when I ran `ant run-IT-HSQLDB`.

- source code change
```
diff --git
a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests
/MockSolrService.java
b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output
/solr/tests/MockSolrService.java
index 237ade09c..3fb558f52 100644
---
a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo
lrService.java
+++
b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo
lrService.java
@@ -18,7 +18,10 @@
 */
 package org.apache.manifoldcf.agents.output.solr.tests;

+import org.eclipse.jetty.http2.server.HTTP2CServerConnectionFactory;
 import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.server.HttpConfiguration;
+import org.eclipse.jetty.server.HttpConnectionFactory;
 import org.eclipse.jetty.server.Server;
 import org.eclipse.jetty.server.ServerConnector;
 import org.eclipse.jetty.servlet.ServletContextHandler;
@@ -40,7 +43,10 @@ public class MockSolrService
   public MockSolrService()
   {
 server = new Server(new QueuedThreadPool(35));
-ServerConnector connector = new ServerConnector(server);
+HttpConfiguration config = new HttpConfiguration();
+HttpConnectionFactory http1 = new HttpConnectionFactory(config);
+HTTP2CServerConnectionFactory http2c = new
HTTP2CServerConnectionFactory(config);
+ServerConnector connector = new ServerConnector(server, http1, http2c);
 connector.setPort(8188);
 server.addConnector(connector);
 servlet = new SolrServlet();
@@ -111,6 +117,7 @@ public class MockSolrService
   res.getWriter().printf(Locale.ROOT, "\n");
   res.getWriter().printf(Locale.ROOT, "\n");
   res.getWriter().flush();
 }
```

- run Junit test
```
~manifoldcf/connectors/solr% ant run-IT-HSQLDB
```

- confirm test-HSQLDB-output/manifoldcf.log
```
ERROR 2023-10-23T22:10:29,902 (Worker thread '24') - Exception tossed:
Unhandled Solr exception during indexing http://test70.txt (200): Error
from server at http://localhost:8188/solr: Expected mime type
application/octet-stream but got application/xml. 
  


org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unhandled Solr
exception during indexing http://test70.txt (200): Error from server at
http://localhost:8188/solr: Expected mime type application/octet-stream but
got application/xml. 
  


at
org.apache.manifoldcf.agents.output.solr.HttpPoster.handleSolrException(HttpPoster.java:389)
~[classes/:?]
at
org.apache.manifoldcf.agents.output.solr.HttpPoster.indexPost(HttpPoster.java:544)
~[classes/:?]
at
org.apache.manifoldcf.agents.output.solr.SolrConnector.addOrReplaceDocumentWithException(SolrConnector.java:522)
~[classes/:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3214)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$OutputAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3395)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3065)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2696)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:750)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1585)
~[mcf-pull-agent.jar:?]
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1550)
~[mcf-pull-agent.jar:?]
at
org.apache.manifoldcf.crawler.tests.TestingRepositoryConnector.processDocuments(TestingRepositoryConnector.java:84)
~[mcf-pull-agent-tests.jar:?]
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:402)
~[mcf-pull-agent.jar:?]
ERROR 2023-10-23T22:10:29,902 (Worker thread '10') - Exception tossed:
Unhandled Solr exception during indexing http://test72.txt (200): Error
from server at http://localhost:8188/solr: Expected mime type
application/octet-stream but got application/xml. 
  

```

Could you give me some advice?

2023年10月23日(月) 22:01 Mingchun Zhao :

> > Then, wherever zookeeper is mentioned in framework/build.xml, a
> reference to those dependencies must also be included.
>
> It looks like zookeeper*.jar was already included in
> connec

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

> Then, wherever zookeeper is mentioned in framework/build.xml, a
reference to those dependencies must also be included.

It looks like zookeeper*.jar was already included in
connector-test-classpath within kafka/build.xml.
```









```


2023年10月23日(月) 21:50 Karl Wright :

> Hi,
>
> That just downloads zookeeper.  But apparently the zookeeper version
> required by Kafka now has dependencies of its own.  Otherwise the zookeeper
> tests wouldn't fail with linkage errors.
>
> The dependencies need to be identified and added in several places.  The
> first place is to the download-zookeeper part of the root build.xml
> script.  Then, wherever zookeeper is mentioned in framework/build.xml, a
> reference to those dependencies must also be included.
>
> Karl
>
>
> On Mon, Oct 23, 2023 at 8:32 AM Mingchun Zhao 
> wrote:
>
> > Hi Karl,
> >
> > > Mingchun, did you add the jar(s) that the new zookeeper needs to the
> > build.xml download section?
> >
> > Are the following settings correct? Or do you have an old version of
> > zookeeper-*.jar left in your environment?
> >
> > ```build.xml
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > ... ...
> > ```
> >
> > Regards,
> > Mingchun
> >
> >
> > 2023年10月23日(月) 21:19 Karl Wright :
> >
> > > Well, that was interesting.
> > >
> > > Didn't get very far, because the dependency updates committed broke
> > > Zookeeper again:
> > >
> > > [junit] Testcase:
> > >
> > >
> >
> multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks):
> > >  Caused an ERROR
> > > [junit] io/netty/handler/ssl/SslContext
> > > [junit] java.lang.NoClassDefFoundError:
> > io/netty/handler/ssl/SslContext
> > > [junit] at
> > >
> > >
> >
> org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106)
> > > [junit] at
> > >
> > >
> >
> org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96)
> > > [junit] at
> > > org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92)
> > > [junit] at
> > > org.apache.zookeeper.common.ZKConfig.(ZKConfig.java:61)
> > > [junit] at
> > >
> org.apache.zookeeper.client.ZKClientConfig.(ZKClientConfig.java:69)
> > > [junit] at
> > > org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:643)
> > > [junit] at
> > > org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:567)
> > > [junit] at
> > > org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:734)
> > > [junit] at
> > > org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:448)
> > > [junit] at
> > >
> > >
> >
> org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74)
> > > [junit] at
> > >
> > >
> >
> org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.(ZooKeeperConnection.java:66)
> > >
> > > It looks like the even newer Zookeeper version has a newer dependency
> > that
> > > isn't being included in the basic classpath, but should be.  Mingchun,
> > did
> > > you add the jar(s) that the new zookeeper needs to the build.xml
> download
> > > section?  If so, can you remind me what they were?
> > >
> > > Karl
> > >
> > >
> > > On Mon, Oct 23, 2023 at 8:11 AM Karl Wright 
> wrote:
> > >
> > > > I begin to suspect that the problem may be human error.
> > > > If you don't do "ant clean-core-deps; ant make-core-deps", but
> instead
> > > > just use "ant make-core-deps", you could have incompatible versions
> of
> > > > several libraries in your classpath for the tests.  I'll try today to
> > > > verify whether that might be happening by trying the tests locally
> > > myself.
> > > >
> > > > Karl
> > > >
> > > >
> > > > On Mon, Oct 23, 2023 at 7:57 AM Guylaine BASSETTE <
> > > > guylaine.basse...@francelabs.com> wrote:
> > > >
> > > >> Hi Karl and Mingchun,
> > > >>
> > > >> Thanks for your work on th

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Hi Karl,

> Mingchun, did you add the jar(s) that the new zookeeper needs to the
build.xml download section?

Are the following settings correct? Or do you have an old version of
zookeeper-*.jar left in your environment?

```build.xml









... ...
```

Regards,
Mingchun


2023年10月23日(月) 21:19 Karl Wright :

> Well, that was interesting.
>
> Didn't get very far, because the dependency updates committed broke
> Zookeeper again:
>
> [junit] Testcase:
>
> multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks):
>  Caused an ERROR
> [junit] io/netty/handler/ssl/SslContext
> [junit] java.lang.NoClassDefFoundError: io/netty/handler/ssl/SslContext
> [junit] at
>
> org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106)
> [junit] at
>
> org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96)
> [junit] at
> org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92)
> [junit] at
> org.apache.zookeeper.common.ZKConfig.(ZKConfig.java:61)
> [junit] at
> org.apache.zookeeper.client.ZKClientConfig.(ZKClientConfig.java:69)
> [junit] at
> org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:643)
> [junit] at
> org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:567)
> [junit] at
> org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:734)
> [junit] at
> org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:448)
> [junit] at
>
> org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74)
> [junit] at
>
> org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.(ZooKeeperConnection.java:66)
>
> It looks like the even newer Zookeeper version has a newer dependency that
> isn't being included in the basic classpath, but should be.  Mingchun, did
> you add the jar(s) that the new zookeeper needs to the build.xml download
> section?  If so, can you remind me what they were?
>
> Karl
>
>
> On Mon, Oct 23, 2023 at 8:11 AM Karl Wright  wrote:
>
> > I begin to suspect that the problem may be human error.
> > If you don't do "ant clean-core-deps; ant make-core-deps", but instead
> > just use "ant make-core-deps", you could have incompatible versions of
> > several libraries in your classpath for the tests.  I'll try today to
> > verify whether that might be happening by trying the tests locally
> myself.
> >
> > Karl
> >
> >
> > On Mon, Oct 23, 2023 at 7:57 AM Guylaine BASSETTE <
> > guylaine.basse...@francelabs.com> wrote:
> >
> >> Hi Karl and Mingchun,
> >>
> >> Thanks for your work on the last few issues. I join you on this Solr
> >> testing problem.
> >>
> >> That said, we've tested this new connector in our application, with a
> >> FileShare job and everything was OK.
> >>
> >> I hope and think it's just a problem specific to the test. Missing
> >> updates or incompatible dependencies...
> >>
> >> Le 20/10/2023 à 02:58, Mingchun Zhao a écrit :
> >> > Hi Karl, Thanks!
> >> >
> >> >> so I wonder if, once again, there's a problem with dependencies for
> the
> >> > version of Solr they chose.
> >> >
> >> > I'll take a look at this issue.
> >> >
> >> > 2023年10月20日(金) 9:50 Karl Wright:
> >> >
> >> >> This connector FranceLabs updated.  The problem seems to occur at a
> >> basic
> >> >> level during http2 communication, so I wonder if, once again,
> there's a
> >> >> problem with dependencies for the version of Solr they chose.
> >> >>
> >> >> Karl
> >> >>
> >> >>
> >> >> On Thu, Oct 19, 2023 at 8:32 PM Mingchun Zhao<
> >> mingchun.zha...@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> About the test "SolrCrawlHSQLDBIT" failure, it seems that "IO
> >> exception
> >> >>> during indexinghttp://test58.txt:
> >> >> frame_size_error/invalid_frame_length"
> >> >>> error is occurring on the ManifoldCF side.
> >> >>>
> >> >>> - command:
> >> >>> ```
> >> >>> manifoldcf/connectors/solr% ant run-IT-HSQLDB
> >> >>>
> >> >>> run-IT-HSQLDB:
> >> >>>  [junit] Testsuite:
> >> >&

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Hi Karl, Thanks!

> so I wonder if, once again, there's a problem with dependencies for the
version of Solr they chose.

I'll take a look at this issue.

2023年10月20日(金) 9:50 Karl Wright :

> This connector FranceLabs updated.  The problem seems to occur at a basic
> level during http2 communication, so I wonder if, once again, there's a
> problem with dependencies for the version of Solr they chose.
>
> Karl
>
>
> On Thu, Oct 19, 2023 at 8:32 PM Mingchun Zhao 
> wrote:
>
> > About the test "SolrCrawlHSQLDBIT" failure, it seems that "IO exception
> > during indexing http://test58.txt:
> frame_size_error/invalid_frame_length"
> > error is occurring on the ManifoldCF side.
> >
> > - command:
> > ```
> > manifoldcf/connectors/solr% ant run-IT-HSQLDB
> >
> > run-IT-HSQLDB:
> > [junit] Testsuite:
> > org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
> > ... ...
> > ```
> >
> > - I checked "connectors/solr/test-HSQLDB-output/manifoldcf.log":
> > ```
> >  WARN 2023-10-20T09:14:56,635 (Worker thread '18') - IO exception during
> > indexing http://test58.txt: frame_size_error/invalid_frame_length
> > java.io.IOException: frame_size_error/invalid_frame_length
> > at org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:566)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at org.eclipse.jetty.http2.HTTP2Session.access$2700(HTTP2Session.java:80)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:1857)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.HTTP2Session$StreamsState.access$400(HTTP2Session.java:1436)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:511)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:506)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onConnectionFailure(Parser.java:414)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.HTTP2Connection$ParserListener.onConnectionFailure(HTTP2Connection.java:397)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:223)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:215)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> org.eclipse.jetty.http2.parser.Parser.connectionFailure(Parser.java:209)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at org.eclipse.jetty.http2.parser.Parser.parseHeader(Parser.java:155)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:121)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:261)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
> > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
> > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
> > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
> > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> > org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:183)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:138)
> > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
> > at
> >
> >
> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.su

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

About the test "SolrCrawlHSQLDBIT" failure, it seems that "IO exception
during indexing http://test58.txt: frame_size_error/invalid_frame_length"
error is occurring on the ManifoldCF side.

- command:
```
manifoldcf/connectors/solr% ant run-IT-HSQLDB

run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
... ...
```

- I checked "connectors/solr/test-HSQLDB-output/manifoldcf.log":
```
 WARN 2023-10-20T09:14:56,635 (Worker thread '18') - IO exception during
indexing http://test58.txt: frame_size_error/invalid_frame_length
java.io.IOException: frame_size_error/invalid_frame_length
at org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:566)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.http2.HTTP2Session.access$2700(HTTP2Session.java:80)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:1857)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session$StreamsState.access$400(HTTP2Session.java:1436)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:511)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:506)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onConnectionFailure(Parser.java:414)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection$ParserListener.onConnectionFailure(HTTP2Connection.java:397)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:223)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:215)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.http2.parser.Parser.connectionFailure(Parser.java:209)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.http2.parser.Parser.parseHeader(Parser.java:155)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:121)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:261)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:183)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:138)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:361)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:69)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:350)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:271)
~[solr-solrj-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
```

2023年10月19日(木) 20:37

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

I'm having trouble with the Solr test `ant run-IT-HSQLDB`.
It's been over an hour since I ran the test and it still doesn't finish.
It seems that the problem is probably due to the Manifold job not finishing.
Do you have any ideas or advice? Thanks!

```
~/ManifoldCF/manifoldcf/connectors/solr% ant run-IT-HSQLDB
Buildfile:
/Users/zhaomingchun/ManifoldCF/manifoldcf/connectors/solr/build.xml

... ...
run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
[junit] Configuration file successfully read
[junit] [main] INFO org.eclipse.jetty.util.log - Logging initialized
@7416ms to org.eclipse.jetty.util.log.Slf4jLog
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0 Scavenging
every 60ms
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@4bdc8b5d{ManifoldCF Crawler
Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1frgn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-17189990303852051874/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-crawler-ui.war}
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@3bcd426c{ManifoldCF Authorities API
Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1frgn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-1541416273384984853/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-authority-service.war}
[junit] Creating mock service
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@5f14a673{ManifoldCF General API
Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1frgn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-12421572948842353797/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-api-service.war}
[junit] Mock service created
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@70325d20{HTTP/1.1, (http/1.1)}{0.0.0.0:8346}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started @9616ms
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0 Scavenging
every 60ms
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.s.ServletContextHandler@7808f638{/solr,null,AVAILABLE}
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@2ab5afc7{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started @9626ms
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Stopped ServerConnector@2ab5afc7{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.session - node0 Stopped
scavenging
[junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Stopped o.e.j.s.ServletContextHandler@7808f638{/solr,null,STOPPED}
```

2023年10月19日(木) 20:05 Mingchun Zhao :

> Hi Karl,
>
> I've tried to update Kafka and its dependencies to the latest version
> including zookeeper,
> and confirmed tha kafka test run-IT-HSQLDB passed as below:
>
> ```
> ~manifoldcf% cd connectors/kafka
> ~manifoldcf/connectors/kafka/% ant run-IT-HSQLDB
>
> BUILD SUCCESSFUL
> Total time: 1 minute 19 seconds
> ```
>
> Also, I prepared a PR for this issue:
> https://github.com/apache/manifoldcf/pull/155
>
> 2023年10月19日(木) 7:12 Karl Wright :
>
>> Hi,
>>
>> It looks like the latest release of Kafka is 3.6.0.
>>
>> I'd try setting that version in the pom for connectors/kafka and doing mvn
>> install.  Then you can see what dependencies it wants by:
>> mvn dependency:tree
>>
>> It may be that Kafka no longer even requires zookeeper - I didn't find it
>> in a cursory inspection. But the dependency:tree would be the final word.
>>
>> Karl
>>
>>
>> On Sat, Oct 14, 2023 at 2:17 AM Mingchun Zhao 
>> wrote:
>>
>> > Kalr, Thanks!
>> > Though 

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Hi Karl,

I've tried to update Kafka and its dependencies to the latest version
including zookeeper,
and confirmed tha kafka test run-IT-HSQLDB passed as below:

```
~manifoldcf% cd connectors/kafka
~manifoldcf/connectors/kafka/% ant run-IT-HSQLDB

BUILD SUCCESSFUL
Total time: 1 minute 19 seconds
```

Also, I prepared a PR for this issue:
https://github.com/apache/manifoldcf/pull/155

2023年10月19日(木) 7:12 Karl Wright :

> Hi,
>
> It looks like the latest release of Kafka is 3.6.0.
>
> I'd try setting that version in the pom for connectors/kafka and doing mvn
> install.  Then you can see what dependencies it wants by:
> mvn dependency:tree
>
> It may be that Kafka no longer even requires zookeeper - I didn't find it
> in a cursory inspection. But the dependency:tree would be the final word.
>
> Karl
>
>
> On Sat, Oct 14, 2023 at 2:17 AM Mingchun Zhao 
> wrote:
>
> > Kalr, Thanks!
> > Though I'm not familiar with kafka, I'll try to find out what's causing
> the
> > error as much as possible.
> >
> > Kind Regards,
> > Mingchun
> >
> >
> > 2023年10月14日(土) 14:07 Karl Wright :
> >
> > > Yes, this seems to be something related to zookeeper update and the
> Kafka
> > > library version we're using.
> > >
> > > Someone will need to dig into what is going wrong here before we can
> > > release.  I don't know how widely used the kafka connector is but if it
> > is
> > > lightly used we can perhaps not distribute the connector any longer.
> But
> > > that would be a last choice.
> > >
> > > Karl
> > >
> > >
> > > On Fri, Oct 13, 2023 at 12:12 PM Mingchun Zhao <
> > mingchun.zha...@gmail.com>
> > > wrote:
> > >
> > > > By applying r1912939, I was able to confirm that the kafka test
> compile
> > > > error has disappeared when running `ant test`.
> > > > Thanks, Karl!
> > > >
> > > > However, the following error occurred on subsequent test runs.
> > > > ```
> > > > [junit] Testsuite:
> > > > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT
> > > > [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> > > elapsed:
> > > > 0 sec
> > > > [junit]
> > > > [junit] Testcase:
> > > >
> > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT:sanityCheck:
> > > >Caused an ERROR
> > > > [junit] Forked Java VM exited abnormally. Please note the time in
> > the
> > > > report does not reflect the time until the VM exit.
> > > > [junit] junit.framework.AssertionFailedError: Forked Java VM
> exited
> > > > abnormally. Please note the time in the report does not reflect the
> > time
> > > > until the VM exit.
> > > > [junit] at
> > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > > > [junit] at
> > > >
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > [junit] at
> > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > > > [junit] at
> > > >
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > [junit] at
> > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > > > [junit] at
> > > >
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > [junit] at
> > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > > > [junit] at
> > > >
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > [junit]
> > > > [junit]
> > > >
> > > > BUILD FAILED
> > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The
> following
> > > > error occurred while executing this line:
> > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The
> following
> > > > error occurred while executing this line:
> > > >
> > /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Kalr, Thanks!
Though I'm not familiar with kafka, I'll try to find out what's causing the
error as much as possible.

Kind Regards,
Mingchun


2023年10月14日(土) 14:07 Karl Wright :

> Yes, this seems to be something related to zookeeper update and the Kafka
> library version we're using.
>
> Someone will need to dig into what is going wrong here before we can
> release.  I don't know how widely used the kafka connector is but if it is
> lightly used we can perhaps not distribute the connector any longer.  But
> that would be a last choice.
>
> Karl
>
>
> On Fri, Oct 13, 2023 at 12:12 PM Mingchun Zhao 
> wrote:
>
> > By applying r1912939, I was able to confirm that the kafka test compile
> > error has disappeared when running `ant test`.
> > Thanks, Karl!
> >
> > However, the following error occurred on subsequent test runs.
> > ```
> > [junit] Testsuite:
> > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT
> > [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed:
> > 0 sec
> > [junit]
> > [junit] Testcase:
> > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT:sanityCheck:
> >Caused an ERROR
> > [junit] Forked Java VM exited abnormally. Please note the time in the
> > report does not reflect the time until the VM exit.
> > [junit] junit.framework.AssertionFailedError: Forked Java VM exited
> > abnormally. Please note the time in the report does not reflect the time
> > until the VM exit.
> > [junit] at
> > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > [junit] at
> >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > [junit] at
> > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > [junit] at
> >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > [junit] at
> > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > [junit] at
> >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > [junit] at
> > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > [junit] at
> >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > [junit]
> > [junit]
> >
> > BUILD FAILED
> > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following
> > error occurred while executing this line:
> > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following
> > error occurred while executing this line:
> > /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102:
> > Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed
> > (crashed)
> > ```
> >
> >
> > 2023年10月13日(金) 21:56 Karl Wright :
> >
> > > r1912939 fixes this but I need to spin a new RC.
> > > Karl
> > >
> > >
> > > On Fri, Oct 13, 2023 at 8:46 AM Karl Wright 
> wrote:
> > >
> > > > Yes I get the same thing; a test needs to be updated.
> > > >
> > > > [javac]
> > > >
> > >
> >
> C:\wip\mcf\release-2.26-branch\connectors\kafka\connector\src\test\java\org\apache\manifoldcf\agents\output\kafka\ZooKeeperLocal.java:45:
> > > > error: unreported exception AdminServerException; must be caught or
> > > > declared to be thrown
> > > > [javac]   zooKeeperServer.runFromConfig(configuration);
> > > > [javac]
> > > >
> > > > Karl
> > > >
> > > >
> > > > On Fri, Oct 13, 2023 at 8:35 AM Karl Wright 
> > wrote:
> > > >
> > > >> There was a Zookeeper dependency change this release.  I wonder if
> > there
> > > >> is a test that needs to be updated.  Let me try and see.
> > > >>
> > > >> Karl
> > > >>
> > > >>
> > > >> On Fri, Oct 13, 2023 at 4:51 AM Piergiorgio Lucidi <
> > > >> piergior...@apache.org> wrote:
> > > >>
> > > >>> Hi Mingchun,
> > > >>>
> > > >>> thank you for your message and I was trying to build ManifoldCF
> using
> > > >>> OpenJDK 17 so probably in the future for supporting this version of
>

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

By applying r1912939, I was able to confirm that the kafka test compile
error has disappeared when running `ant test`.
Thanks, Karl!

However, the following error occurred on subsequent test runs.
```
[junit] Testsuite:
org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
0 sec
[junit]
[junit] Testcase:
org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT:sanityCheck:
   Caused an ERROR
[junit] Forked Java VM exited abnormally. Please note the time in the
report does not reflect the time until the VM exit.
[junit] junit.framework.AssertionFailedError: Forked Java VM exited
abnormally. Please note the time in the report does not reflect the time
until the VM exit.
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit]
[junit]

BUILD FAILED
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102:
Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed
(crashed)
```


2023年10月13日(金) 21:56 Karl Wright :

> r1912939 fixes this but I need to spin a new RC.
> Karl
>
>
> On Fri, Oct 13, 2023 at 8:46 AM Karl Wright  wrote:
>
> > Yes I get the same thing; a test needs to be updated.
> >
> > [javac]
> >
> C:\wip\mcf\release-2.26-branch\connectors\kafka\connector\src\test\java\org\apache\manifoldcf\agents\output\kafka\ZooKeeperLocal.java:45:
> > error: unreported exception AdminServerException; must be caught or
> > declared to be thrown
> > [javac]   zooKeeperServer.runFromConfig(configuration);
> > [javac]
> >
> > Karl
> >
> >
> > On Fri, Oct 13, 2023 at 8:35 AM Karl Wright  wrote:
> >
> >> There was a Zookeeper dependency change this release.  I wonder if there
> >> is a test that needs to be updated.  Let me try and see.
> >>
> >> Karl
> >>
> >>
> >> On Fri, Oct 13, 2023 at 4:51 AM Piergiorgio Lucidi <
> >> piergior...@apache.org> wrote:
> >>
> >>> Hi Mingchun,
> >>>
> >>> thank you for your message and I was trying to build ManifoldCF using
> >>> OpenJDK 17 so probably in the future for supporting this version of
> Java
> >>> we
> >>> should include Jaxb libraries as well.
> >>>
> >>> The build is ok now and I can compile and package everything correctly.
> >>> Unfortunately executing tests I have the following error:
> >>>
> >>> compile-tests:
> >>> [javac] Compiling 1 source file to
> >>>
> >>>
> /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/build/connector-tests/classes
> >>> [javac]
> >>>
> >>>
> /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/connector/src/test/java/org/apache/manifoldcf/agents/output/kafka/ZooKeeperLocal.java:45:
> >>> error: unreported exception AdminServerException; must be caught or
> >>> declared to be thrown
> >>> [javac]   zooKeeperServer.runFromConfig(configuration);
> >>> [javac]^
> >>> [javac] 1 error
> >>>
> >>> BUILD FAILED
> >>>
> /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:497:
> >>> The following error occurred while executing this line:
> >>>
> /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:471:
> >>> The following error occurred while executing this line:
> >>>
> >>>
> /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/dist/connector-build.xml:720:
>

Re: [VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Hi Piergiorgio,

FYI, Allow me to share the java and ant versions and the build steps I ran
in my environment.

```
$ java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed
mode)

$ ant -version
Apache Ant(TM) version 1.10.0 compiled on December 27 2016
```
```
ant clean
ant clean-deps
ant clean-core-deps
ant make-core-deps
ant make-deps
ant build
```

Regards,
Mingchun

2023年10月12日(木) 17:32 Piergiorgio Lucidi :

> Hi folks,
>
> it seems that I can't compile the CswsConnector:
>
>[javac] public List getAttributeGroups()
> [javac]   ^
> [javac]   symbol:   class AttributeGroup
> [javac]   location: class CswsConnector.ObjectInformation
> [javac]
>
> /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:3966:
> error: cannot find symbol
> [javac] public NodePermissions getPermissions()
> [javac]^
> [javac]   symbol:   class NodePermissions
> [javac]   location: class CswsConnector.ObjectInformation
> [javac] 100 errors
> [javac] 1 warning
> [javac] only showing the first 100 errors, of 123 total; use -Xmaxerrs
> if you would like to see more
>
> BUILD FAILED
> /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:489:
> The following error occurred while executing this line:
> /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:471:
> The following error occurred while executing this line:
>
> /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/dist/connector-build.xml:686:
> Compile failed; see the compiler error output for details.
>
> Do you have any ideas?
>
> Thanks,
> PG
>
> Il giorno gio 12 ott 2023 alle ore 09:03 Guylaine BASSETTE <
> guylaine.basse...@francelabs.com> ha scritto:
>
> > Hi,
> >
> > +1 from France Labs
> >
> > Regards,
> > Guylaine
> >
> > France Labs – Your knowledge, now
> > Datafari Enterprise Search – Découvrez la version 5 / Discover our
> version
> > 5
> > www.datafari.com <http://www.datafari.com>
> > Le 11/10/2023 à 23:12, Furkan KAMACI a écrit :
> > > Hi,
> > >
> > > +1!
> > >
> > > Kind regards,
> > > Furkan Kamaci
> > >
> > > On Wed, Oct 11, 2023 at 11:13 PM Mingchun Zhao<
> mingchun.zha...@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi,
> > >> Ran all the tests.
> > >> +1 from me.
> > >>
> > >> Kind regards,
> > >> Mingchun
> > >>
> > >> 2023年10月12日(木) 4:29 Cihad Guzel:
> > >>
> > >>> Hi,
> > >>>
> > >>> +1 from me
> > >>>
> > >>> Regards,
> > >>> Cihad Guzel
> > >>>
> > >>> 11 Eki 2023 Çar, saat 15:08 tarihinde Karl Wright >
> > >>> şunu
> > >>> yazdı:
> > >>>
> > >>>> Please vote on whether to release Apache ManifoldCF 2.26, RC0.
> > >>>> This release is the first release that requires at least Java 11,
> and
> > >> it
> > >>>> also includes a new CSV connector along with support for Solr 9.
> The
> > >>>> release artifact can be found at:
> > >>>>
> > >>
> > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.26
> > >>> ,
> > >>>> and there is a release tag also at
> > >>>> https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC0
> .
> > >>>>
> > >>>> Karl
> > >>>>
>
>
>
> --
> Piergiorgio
>

Re: [VOTE] Release ManifoldCF 2.26, RC0

[Mingchun Zhao]

Hi,
Ran all the tests.
+1 from me.

Kind regards,
Mingchun

2023年10月12日(木) 4:29 Cihad Guzel :

> Hi,
>
> +1 from me
>
> Regards,
> Cihad Guzel
>
> 11 Eki 2023 Çar, saat 15:08 tarihinde Karl Wright 
> şunu
> yazdı:
>
> > Please vote on whether to release Apache ManifoldCF 2.26, RC0.
> > This release is the first release that requires at least Java 11, and it
> > also includes a new CSV connector along with support for Solr 9.  The
> > release artifact can be found at:
> > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.26
> ,
> > and there is a release tag also at
> > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC0 .
> >
> > Karl
> >
>

[jira] [Comment Edited] (CONNECTORS-1748) Add "User-Agent platform" option for crawling mobile websites

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773001#comment-17773001
 ] 

Mingchun Zhao edited comment on CONNECTORS-1748 at 10/11/23 7:56 PM:
-

Hi [~daddywri] ,

I've prepared a PR as below:
https://github.com/apache/manifoldcf/pull/154
Could you confirm this please?

Best regards,
Mingchun


was (Author: mingchun.zhao):
Hi [~daddywri] , 

I've prepared a PR as below:
[https://github.com/apache/manifoldcf/pull/153]
Could you confirm this please?

Best regards,
Mingchun

> Add "User-Agent platform" option for crawling mobile websites
> -
>
> Key: CONNECTORS-1748
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1748
> Project: ManifoldCF
>  Issue Type: New Feature
>    Reporter: Mingchun Zhao
>Priority: Major
>
> Hi [~daddywri] ,
> Allow me propose a new feature for crawling mobile sites which restrict 
> access to content based on User-Agent header.
> Actually, Our customer's mobile website was failing to crawl because access 
> was restricted based on whether the User-Agent request header includes the 
> mobile info. For this reason, we added the "User-Agent platform" option to 
> the new "Request Headers" tab on the web repository connector page so that 
> this mobile website doesn't fail to crawl.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Propose adding "User-Agent platform" option for crawling mobile sites

[Mingchun Zhao]

Hi all,

Allow me to propose a new feature for crawling mobile sites that restrict
access to content based on User-Agent header.
Actually, Our customer's mobile website was failing to crawl because access
was restricted based on whether the User-Agent request header is related to
the mobile platform.
For this reason, we added the "User-Agent platform" option to the new
"Request Headers" tab on the web repository connector page so that this
mobile website doesn't fail to crawl.

I've created a jira issue and PR if you are interested:
https://issues.apache.org/jira/projects/CONNECTORS/issues/CONNECTORS-1748
https://github.com/apache/manifoldcf/pull/153

Kind regards,
Mingchun

Re: Ready for the 2.26 release?

[Mingchun Zhao]

Hi Karl, Olivier, Guylaine,

> It looks like the native2ascii invocations have been stripped entirely
from
the build.xml files everywhere, not just in ui-core.

FYI, the native2ascii command disappeared in the patch I provided to
support Java 11 which was included in the PR below. I'm so sorry for that.
https://github.com/apache/manifoldcf/pull/150/files#diff-38e7ea3183260ebac5d6772723b64a3b69ce8aab6a0c1ca2ada6fd9f410c0ba6L283

Thank you for investigating this matter!

Regards,
Mingchun

2023年10月10日(火) 2:17 Karl Wright :

> It's worse than that.
> It looks like the native2ascii invocations have been stripped entirely from
> the build.xml files everywhere, not just in ui-core.  I'm trying to figure
> out when that happened and see if I can get it back.
>
> Karl
>
>
> On Mon, Oct 9, 2023 at 12:42 PM Olivier Tavard <
> olivier.tav...@francelabs.com> wrote:
>
> > Hi Karl,
> >
> > My colleague Guylaine noticed that one line was missing into
> > framework/build.xml into the latest version :
> >  > dest="build/ui-core/classes" includes="**/*.properties" />
> >
> > So all the i18n translation files were missing into the build. The
> correct
> > code is :
> >
> > 
> > 
> >  > destdir="build/ui-core/classes" deprecation="true" target="1.8"
> > source="1.8" debug="true" encoding="utf-8"
> debuglevel="lines,vars,source">
> > 
> > 
> > 
> > 
> > 
> >   *   > src="ui-core/src/main/native2ascii" dest="build/ui-core/classes"
> > includes="**/*.properties" />*
> > 
> >
> > By adding this line, the build  is now OK. We can do a PR tomorrow for
> > that if needed.
> >
> > Thanks,
> >
> >
> > Olivier TAVARD
> > Directeur Général - Cofondateur
> > France Labs – Makers of Datafari Enteprise Search
> > Datafari Enterprise Search 
> > 
> >
> > [image:
> > image003.jpg]
> >
> > Le 9 oct. 2023 à 18:32, Karl Wright  a écrit :
> >
> > I looked very briefly at this and discovered that every message is
> > affected.
> > It may be due, therefore, to jetty refusing access to the translation
> > resources.  But if that's the case I'm not going to be able to do
> anything
> > to get this release out this month; I'm booked solid in fact until
> January.
> >
> > So good luck, folks.  I'd try rolling the Jetty version update back if
> you
> > can as a first step.
> >
> > Karl
> >
> >
> > On Mon, Oct 9, 2023 at 9:08 AM Karl Wright  wrote:
> >
> > No change to paths has been made.
> > Probably the translation files have been corrupted due to many merges and
> > perhaps bad encodings for some of them.  It will need to be looked into.
> > Karl
> >
> >
> > On Mon, Oct 9, 2023 at 8:40 AM Guylaine BASSETTE <
> > guylaine.basse...@francelabs.com> wrote:
> >
> > Hello all,
> >
> > We have tested this version and everything is OK, except for translation,
> > something is broken:
> >
> > It looks like the translations files are not found anymore. Maybe a
> > change in the path to those files ?
> > Best regards,
> > Guylaine
> >
> > France Labs – Your knowledge, now
> > Datafari Enterprise Search – Découvrez la version 5 / Discover our
> > version 5
> > www.datafari.com
> >
> > Retrouvez-nous à Milipol  du 14 au 17 novembre
> >
> > Le 06/10/2023 à 09:31, Karl Wright a écrit :
> >
> > Hi all,
> >
> > The tentative release schedule had a release going out on Sept 30th,
> which
> > is now overdue.  Partly this was because of me, but also partly it's the
> > result of new contributions from France Labs.  But these contributions
> are
> > now committed to trunk and we could go ahead - unless others are expected
> > to be coming shortly, in which case we should wait.  Please let me know.
> >
> > In any case, if I don't hear back by this weekend I will try to create a
> > release candidate then.
> >
> > Karl
> >
> >
> >
>

[jira] [Assigned] (CONNECTORS-1748) Add "User-Agent platform" option for crawling mobile websites

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao reassigned CONNECTORS-1748:
-

Assignee: (was: Mingchun Zhao)

> Add "User-Agent platform" option for crawling mobile websites
> -
>
> Key: CONNECTORS-1748
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1748
> Project: ManifoldCF
>  Issue Type: New Feature
>    Reporter: Mingchun Zhao
>Priority: Major
>
> Hi [~daddywri] ,
> Allow me propose a new feature for crawling mobile sites which restrict 
> access to content based on User-Agent header.
> Actually, Our customer's mobile website was failing to crawl because access 
> was restricted based on whether the User-Agent request header includes the 
> mobile info. For this reason, we added the "User-Agent platform" option to 
> the new "Request Headers" tab on the web repository connector page so that 
> this mobile website doesn't fail to crawl.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1748) Add "User-Agent platform" option for crawling mobile websites

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773001#comment-17773001
 ] 

Mingchun Zhao commented on CONNECTORS-1748:
---

Hi [~daddywri] , 

I've prepared a PR as below:
[https://github.com/apache/manifoldcf/pull/153]
Could you confirm this please?

Best regards,
Mingchun

> Add "User-Agent platform" option for crawling mobile websites
> -
>
> Key: CONNECTORS-1748
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1748
> Project: ManifoldCF
>  Issue Type: New Feature
>    Reporter: Mingchun Zhao
>Assignee: Mingchun Zhao
>Priority: Major
>
> Hi [~daddywri] ,
> Allow me propose a new feature for crawling mobile sites which restrict 
> access to content based on User-Agent header.
> Actually, Our customer's mobile website was failing to crawl because access 
> was restricted based on whether the User-Agent request header includes the 
> mobile info. For this reason, we added the "User-Agent platform" option to 
> the new "Request Headers" tab on the web repository connector page so that 
> this mobile website doesn't fail to crawl.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (CONNECTORS-1748) Add "User-Agent platform" option for crawling mobile websites

[Mingchun Zhao (Jira)]

Mingchun Zhao created CONNECTORS-1748:
-

 Summary: Add "User-Agent platform" option for crawling mobile 
websites
 Key: CONNECTORS-1748
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1748
 Project: ManifoldCF
  Issue Type: New Feature
Reporter: Mingchun Zhao
Assignee: Mingchun Zhao


Hi [~daddywri] ,
Allow me propose a new feature for crawling mobile sites which restrict access 
to content based on User-Agent header.
Actually, Our customer's mobile website was failing to crawl because access was 
restricted based on whether the User-Agent request header includes the mobile 
info. For this reason, we added the "User-Agent platform" option to the new 
"Request Headers" tab on the web repository connector page so that this mobile 
website doesn't fail to crawl.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: branches/CONNECTORS-1740

[Mingchun Zhao]

Hi Karl,

Thanks! We might need to update the ant version as well as the JDK version.
When I build with trunk in my environment, the build passes when I use
ant1.10.0 or ant1.10.13, but the following build error occurs when I use
ant1.8.2 or ant1.9.0.
==
BUILD FAILED
/manifoldcf/build.xml:294: The following error occurred while executing
this line:
/manifoldcf/framework/build.xml:283: Error starting Sun's native2ascii:
==

Regards,
Mingchun

2023年6月7日(水) 19:34 Karl Wright :

> That's because I changed the method call to include a "null" argument that
> it didn't have before.
>
> If all the versions are right I think that is all we need to do.
>
> FWIW, the change does require JDK 11 and if you don't have your JAVA_HOME
> set up to point to that it won't build correctly now.
>
>
> On Tue, Jun 6, 2023 at 6:42 PM Mingchun Zhao 
> wrote:
>
> > Hi Karl,
> >
> > Thanks for your reply!
> >
> > I pulled the latest trunk and confirmed the jetty.version within
> build.xml
> > and pom.xml was already updated to "9.4.48.v20220622".
> >
> > And then, I tried the build command:
> >
> > *ant clean clean-deps clean-core-deps make-core-deps make-deps build*
> >
> > with different combinations of jdk and ant versions and got different
> > results.
> > However, the build error below as you pointed out in your previous mail
> did
> > not occur.
> >
> > *incompatible types: HttpClientTransport cannot be converted to
> > SslContextFactory*
> >
> > ## jdk11.0.11 + ant1.10.13
> >
> > ==
> >
> >
> >
> > *BUILD SUCCESSFULTotal time: 1 minute 52 seconds==*
> > ## jdk11.0.11 + ant1.8.2
> >
> > ==
> > *BUILD FAILED*
> > */Users/zhaomingchun/mcf/manifoldcf/build.xml:294: The following error
> > occurred while executing this line:*
> > */Users/zhaomingchun/mcf/manifoldcf/framework/build.xml:283: Error
> starting
> > Sun's native2ascii:*
> > *Total time: 5 seconds*
> > ==
> >
> > ## jdk1.8.0_292 + ant1.8.2(Also ant1.10.13)
> > *==*
> > *compile-connector:*
> > * [javac] Compiling 17 source files to
> >
> /Users/zhaomingchun/mcf/manifoldcf/connectors/solr/build/connector/classes*
> > * [javac]
> >
> >
> /Users/zhaomingchun/mcf/manifoldcf/connectors/solr/connector/src/main/java/org/apache/manifoldcf/agents/output/solr/ModifiedHttp2SolrClient.java:3:
> > error: cannot access Utils*
> > * [javac] import static
> > org.apache.solr.common.util.Utils.getObjectByPath;*
> > * [javac] ^*
> > * [javac] bad class file:
> >
> >
> /Users/zhaomingchun/mcf/manifoldcf/lib/solr-solrj-9.1.0.jar(org/apache/solr/common/util/Utils.class)*
> > * [javac] class file has wrong version 55.0, should be 52.0*
> > * [javac] Please remove or make sure it appears in the correct
> > subdirectory of the classpath.*
> >
> > *BUILD FAILED*
> > */Users/zhaomingchun/mcf/manifoldcf/build.xml:487: The following error
> > occurred while executing this line:*
> > */Users/zhaomingchun/mcf/manifoldcf/build.xml:469: The following error
> > occurred while executing this line:*
> > */Users/zhaomingchun/mcf/manifoldcf/dist/connector-build.xml:686: Compile
> > failed; see the compiler error output for details.*
> >
> > *Total time: 5 minutes 27 seconds*
> > ==
> >
> > Could you please tell me which java and ant version you are using?
> >
> > Regards,
> > Mingchun
> >
> >
> > 2023年6月7日(水) 6:10 Karl Wright :
> >
> > > Interestingly I updated trunk by merging the branch, so I would have
> > > expected the Jetty update to have happened properly.  You may want to
> > check
> > > why it didn't.
> > >
> > > Karl
> > >
> > >
> > > On Tue, Jun 6, 2023 at 4:29 PM Mingchun Zhao <
> mingchun.zha...@gmail.com>
> > > wrote:
> > >
> > > > Hi Karl,
> > > >
> > > > Thanks for your email. About this compile error, I think there are
> two
> > > > ways to fix it.
> > > >
> > > > 1. Change jetty.version within build.xml and pom.xml to
> > > > 9.4.48.v20220622 same as CONNECTORS-1740 branch:
> > > >
> > > > build.xml
> > > > - 
> > > > + 
> > > >
> > > > pom.xml
> > > > - 9.4.25.v20191220
> > > > + 9.4.48.v20220622
> > > >
> > > > 2. Change the parameters of the HttpClient function within
> > > > ModifiedHttp2SolrClient.java as below:
> > &g

Re: branches/CONNECTORS-1740

[Mingchun Zhao]

Hi Karl,

Thanks for your reply!

I pulled the latest trunk and confirmed the jetty.version within build.xml
and pom.xml was already updated to "9.4.48.v20220622".

And then, I tried the build command:

*ant clean clean-deps clean-core-deps make-core-deps make-deps build*

with different combinations of jdk and ant versions and got different
results.
However, the build error below as you pointed out in your previous mail did
not occur.

*incompatible types: HttpClientTransport cannot be converted to
SslContextFactory*

## jdk11.0.11 + ant1.10.13

==



*BUILD SUCCESSFULTotal time: 1 minute 52 seconds==*
## jdk11.0.11 + ant1.8.2

==
*BUILD FAILED*
*/Users/zhaomingchun/mcf/manifoldcf/build.xml:294: The following error
occurred while executing this line:*
*/Users/zhaomingchun/mcf/manifoldcf/framework/build.xml:283: Error starting
Sun's native2ascii:*
*Total time: 5 seconds*
==

## jdk1.8.0_292 + ant1.8.2(Also ant1.10.13)
*==*
*compile-connector:*
* [javac] Compiling 17 source files to
/Users/zhaomingchun/mcf/manifoldcf/connectors/solr/build/connector/classes*
* [javac]
/Users/zhaomingchun/mcf/manifoldcf/connectors/solr/connector/src/main/java/org/apache/manifoldcf/agents/output/solr/ModifiedHttp2SolrClient.java:3:
error: cannot access Utils*
* [javac] import static
org.apache.solr.common.util.Utils.getObjectByPath;*
* [javac] ^*
* [javac] bad class file:
/Users/zhaomingchun/mcf/manifoldcf/lib/solr-solrj-9.1.0.jar(org/apache/solr/common/util/Utils.class)*
* [javac] class file has wrong version 55.0, should be 52.0*
* [javac] Please remove or make sure it appears in the correct
subdirectory of the classpath.*

*BUILD FAILED*
*/Users/zhaomingchun/mcf/manifoldcf/build.xml:487: The following error
occurred while executing this line:*
*/Users/zhaomingchun/mcf/manifoldcf/build.xml:469: The following error
occurred while executing this line:*
*/Users/zhaomingchun/mcf/manifoldcf/dist/connector-build.xml:686: Compile
failed; see the compiler error output for details.*

*Total time: 5 minutes 27 seconds*
==

Could you please tell me which java and ant version you are using?

Regards,
Mingchun


2023年6月7日(水) 6:10 Karl Wright :

> Interestingly I updated trunk by merging the branch, so I would have
> expected the Jetty update to have happened properly.  You may want to check
> why it didn't.
>
> Karl
>
>
> On Tue, Jun 6, 2023 at 4:29 PM Mingchun Zhao 
> wrote:
>
> > Hi Karl,
> >
> > Thanks for your email. About this compile error, I think there are two
> > ways to fix it.
> >
> > 1. Change jetty.version within build.xml and pom.xml to
> > 9.4.48.v20220622 same as CONNECTORS-1740 branch:
> >
> > build.xml
> > - 
> > + 
> >
> > pom.xml
> > - 9.4.25.v20191220
> > + 9.4.48.v20220622
> >
> > 2. Change the parameters of the HttpClient function within
> > ModifiedHttp2SolrClient.java as below:
> >
> > - httpClient = sslEnabled ? new HttpClient(transport,
> > sslContextFactory) : new HttpClient(transport);
> > + httpClient = sslEnabled ? new HttpClient(transport,
> > sslContextFactory) : new HttpClient(transport, null);
> >
> > The reason for this fix is that the constructor
> > HttpClient(HttpClientTransport) does not exist in older jetty.version
> > like 9.4.25.v20191220, so it seems that trying to use the constructor
> > HttpClient(SslContextFactory) caused a conversion error.
> >
> >
> >
> https://www.javadoc.io/doc/org.eclipse.jetty/jetty-project/9.4.25.v20191220/org/eclipse/jetty/client/HttpClient.html#%3Cinit%3E(org.eclipse.jetty.client.HttpClientTransport,org.eclipse.jetty.util.ssl.SslContextFactory)
> >
> > Best Regards,
> > Mingchun
> >
> > 2023年6月6日(火) 10:03 Karl Wright :
> > >
> > > Hi Mingchun,
> > >
> > > The previous work done on this branch is almost complete but there is
> > still
> > > a build error I get:
> > >
> > > [javac]
> > >
> >
> C:\wip\mcf\trunk\connectors\solr\connector\src\main\java\org\apache\manifoldcf\agents\output\solr\ModifiedHttp2SolrClient.java:200:
> > > error: incompatible types: HttpClientTransport cannot be converted to
> > > SslContextFactory
> > > [javac]   httpClient = sslEnabled ? new HttpClient(transport,
> > > sslContextFactory) : new HttpClient(transport);
> > > [javac]
> > >^
> > >
> > > This didn't show up until I merged the branch onto trunk.  I haven't
> yet
> > > committed it because it doesn't quite build.  Any idea how to resolve
> > this?
> > >
> > > Karl
> >
>

Re: branches/CONNECTORS-1740

[Mingchun Zhao]

Hi Karl,

Thanks for your email. About this compile error, I think there are two
ways to fix it.

1. Change jetty.version within build.xml and pom.xml to
9.4.48.v20220622 same as CONNECTORS-1740 branch:

build.xml
- 
+ 

pom.xml
- 9.4.25.v20191220
+ 9.4.48.v20220622

2. Change the parameters of the HttpClient function within
ModifiedHttp2SolrClient.java as below:

- httpClient = sslEnabled ? new HttpClient(transport,
sslContextFactory) : new HttpClient(transport);
+ httpClient = sslEnabled ? new HttpClient(transport,
sslContextFactory) : new HttpClient(transport, null);

The reason for this fix is that the constructor
HttpClient(HttpClientTransport) does not exist in older jetty.version
like 9.4.25.v20191220, so it seems that trying to use the constructor
HttpClient(SslContextFactory) caused a conversion error.

https://www.javadoc.io/doc/org.eclipse.jetty/jetty-project/9.4.25.v20191220/org/eclipse/jetty/client/HttpClient.html#%3Cinit%3E(org.eclipse.jetty.client.HttpClientTransport,org.eclipse.jetty.util.ssl.SslContextFactory)

Best Regards,
Mingchun

2023年6月6日(火) 10:03 Karl Wright :
>
> Hi Mingchun,
>
> The previous work done on this branch is almost complete but there is still
> a build error I get:
>
> [javac]
> C:\wip\mcf\trunk\connectors\solr\connector\src\main\java\org\apache\manifoldcf\agents\output\solr\ModifiedHttp2SolrClient.java:200:
> error: incompatible types: HttpClientTransport cannot be converted to
> SslContextFactory
> [javac]   httpClient = sslEnabled ? new HttpClient(transport,
> sslContextFactory) : new HttpClient(transport);
> [javac]
>^
>
> This didn't show up until I merged the branch onto trunk.  I haven't yet
> committed it because it doesn't quite build.  Any idea how to resolve this?
>
> Karl

[jira] [Commented] (CONNECTORS-1740) Solr 9 output connector

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17729126#comment-17729126
 ] 

Mingchun Zhao commented on CONNECTORS-1740:
---

Hi [~kwri...@metacarta.com] [~julienFL] ,

I’ve tried to modified the source code for using Zookeeper `3.5.6` and 
migrating Java from 8 to `11`, and the test run-tests-framework was OK. 
Attached the patch [^CONNECTORS-1740.patch] I used in my test.

Also, I tried various other Zookeeper versions(3.6.2, 3.7.0, 3.7.2, 3.8.0, 
3.8.1), but all of them failed in the test run-tests-framework due to an error 
as below.

```
[junit] [reader] INFO org.apache.zookeeper.ZooKeeper - Initiating client 
connection, connectString=localhost:8348 sessionTimeout=2000 
watcher=org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection$ZooKeeperWatcher@b647be3
 
[junit] [reader] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer 
value is 1048575 Bytes 
[junit] [reader] INFO org.apache.zookeeper.ClientCnxn - 
zookeeper.request.timeout value is 0. feature enabled=false 
[junit] [reader-SendThread(localhost:8348)] INFO 
org.apache.zookeeper.ClientCnxn - Opening socket connection to server 
localhost/0:0:0:0:0:0:0:1:8348. 
[junit] [reader-SendThread(localhost:8348)] INFO 
org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to 
authenticate using SASL (unknown error) 
[junit] [reader-SendThread(localhost:8348)] WARN 
org.apache.zookeeper.ClientCnxn - Session 0x0 for sever 
localhost/0:0:0:0:0:0:0:1:8348, Closing socket connection. Attempting reconnect 
except it is a SessionExpiredException. 
[junit] java.net.ConnectException: Connection refused 
[junit] at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
[junit] at 
java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
 
[junit] at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
 
[junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1280)
```

> Solr 9 output connector
> ---
>
> Key: CONNECTORS-1740
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1740
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Lucene/SOLR connector
>Affects Versions: ManifoldCF 2.23
>Reporter: Julien Massiera
>Assignee: Julien Massiera
>Priority: Major
> Attachments: CONNECTORS-1740.patch
>
>
> The current Solr output connector is not compatible with Solr 9.x
> We need to update the connector with SolrJ 9 and make sure that the custom 
> code (multipart post requests, basic/preemptive auth) is still required, and, 
> in case it is, port it ! 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1740) Solr 9 output connector

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1740:
--
Attachment: CONNECTORS-1740.patch

> Solr 9 output connector
> ---
>
> Key: CONNECTORS-1740
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1740
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Lucene/SOLR connector
>Affects Versions: ManifoldCF 2.23
>Reporter: Julien Massiera
>Assignee: Julien Massiera
>Priority: Major
> Attachments: CONNECTORS-1740.patch
>
>
> The current Solr output connector is not compatible with Solr 9.x
> We need to update the connector with SolrJ 9 and make sure that the custom 
> code (multipart post requests, basic/preemptive auth) is still required, and, 
> in case it is, port it ! 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release ManifoldCF 2.25, RC0

[Mingchun Zhao]

+1 (non-binding)

The following tests passed.
- Unit tests
- Integration tests with PostgreSQL
- Load tests with PostgreSQL
- New feature: the ability to disable hopcount tracking entirely, for
better performance of the web connector

Regards,
Mingchun

2023年5月30日(火) 6:08 Karl Wright :
>
> Please vote on whether to release ManifoldCF 2.25, RC0.
>
> This release contains one new feature: the ability to disable hopcount
> tracking entirely, for better performance of the web connector.  The
> attempt to update the Solr connector to release 9.x of Solr did NOT make it
> in because that version of SolrJ depends on a broken version of zookeeper,
> our thread coordination library.
>
> A release artifact can be found here:
> https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.25
>
> A release tag can also be found at
> https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.25-RC0 .
>
> Karl

[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726850#comment-17726850
 ] 

Mingchun Zhao commented on CONNECTORS-1747:
---

[~kwri...@metacarta.com] Thanks, that's so helpful!

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.25
>
> Attachments: CONNECTORS-1747.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726777#comment-17726777
 ] 

Mingchun Zhao commented on CONNECTORS-1747:
---

[~kwri...@metacarta.com], If you need my help with the release work, I'll do 
whatever I can, so feel free to ask me please. I feel sorry for not being able 
to participate in ManifoldCF community activities until now, but I will be 
actively involved in MCF activities from now on. Also, I will try to contact 
committers in Japan to liven up ManifoldCF together. Thank you.

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.25
>
> Attachments: CONNECTORS-1747.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725971#comment-17725971
 ] 

Mingchun Zhao commented on CONNECTORS-1747:
---

Hi [~kwri...@metacarta.com],

Thank you for confirming this patch. 
BTW, do we have a plan for when the next release of ManifoldCF will be? The 
project I'm participating in now uses ManifoldCF+PostgreSQL and will be 
released at the end of next month. It would be very helpful if the latest 
version of MCF could be used. If there is anything I can do, I will actively 
participate in the MCF release work.

Kind regards,
Mingchun

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Assignee: Karl Wright
>Priority: Major
> Attachments: CONNECTORS-1747.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725478#comment-17725478
 ] 

Mingchun Zhao commented on CONNECTORS-1747:
---

Hi [~kwri...@metacarta.com],

I've created a patch as you mentioned above, could you please confirm the 
attached [^CONNECTORS-1747.patch]
In my testing with the attached patch, when I set the additional property as 
below,
``
I confirmed:
(1) The hopcount handling was completely disabled.
(2) No records were inserted into the `intrinsiclink` or `hopcount` tables.
(3) The hopcount tab did not appear in the UI for any job. 

Regards,
Mingchun

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Assignee: Karl Wright
>Priority: Major
> Attachments: CONNECTORS-1747.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1747:
--
Attachment: CONNECTORS-1747.patch

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Assignee: Karl Wright
>Priority: Major
> Attachments: CONNECTORS-1747.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1747:
--
Attachment: (was: JobManager.java.patch)

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Assignee: Karl Wright
>Priority: Major
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724663#comment-17724663
 ] 

Mingchun Zhao commented on CONNECTORS-1747:
---

[~kwri...@metacarta.com] Thank you for your review, it was very helpful. I 
understood, will try and fix the patch as you mentioned above.

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Assignee: Karl Wright
>Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: About disabling hopcount tracking

[Mingchun Zhao]

Hi Karl,

I am terribly sorry for bothering you while you are busy.
For this issue, I've tried to add a property for disabling hopcount
logging to the database only for jobs with its hopcount mode set to
"keep unreachable documents, forever" in the "Hop Filters" tab.
I would appreciate it if you could give me your opinion or advice.


https://issues.apache.org/jira/browse/CONNECTORS-1747

Kind regards,
Mingchun

2023年5月11日(木) 15:28 Mingchun Zhao :
>
> Hi Karl,
>
> Thank you for taking time out of your busy schedule to reply.
>
> > There is an option on the "hopcount" tab of your job to disable hopcount
>
> You mean setting "Hop count mode" to "keep unreachable documents,
> forever" in the "Hop Filters" tab?
> Yes, I did it, however, it seems that the records were still inserted
> into the "intrinsiclink" and "hopcount" tables. Is there a way to tell
> MCF not to insert data into those tables because operations on it can
> become a performance bottleneck when the tables bloat?
>
> Regards,
> Mingchun
>
> 2023年5月10日(水) 19:53 Karl Wright :
> >
> > There is an option on the "hopcount" tab of your job to disable hopcount
> > tracking entirely.
> > Karl
> >
> > On Tue, May 9, 2023 at 11:49 PM Mingchun Zhao 
> > wrote:
> >
> > > Hi Karl,
> > >
> > > Could you please advise me on tracking hopcount.
> > > I'm using ManifoldCF 2.24 with PostgreSQL 12.14 as the database for now.
> > > In my case, I don't need to use the 'Hop Filters' feature so I'd like
> > > to disable tracking hopcount and reduce the insert/update/delete load
> > > on the 'intrinsiclink' and 'hopcount' tables. So I have two questions
> > > about this.
> > > First, is there an option to disable tracking hopcount?
> > > Second, if I disable tracking hopcount , can it affect other crawling
> > > processes?
> > >
> > > Thank you in advance.
> > > Kind regards,
> > > Mingchun
> > >

[jira] [Updated] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1747:
--
Attachment: JobManager.java.patch

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1747:
--
Attachment: (was: JobManager.java.patch)

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1747:
--
Attachment: JobManager.java.patch

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723328#comment-17723328
 ] 

Mingchun Zhao edited comment on CONNECTORS-1747 at 5/17/23 3:18 AM:


Hello,

I changed spec for new property as below. Could you please review the attached 
new patch?



You can use this property to disable logging hopcount to database only for jobs 
with hopcount mode "keep unreachable documents, forever" specified in the "Hop 
Filters" tab.


was (Author: mingchun.zhao):
Hello,

I changed spec for new property as below. Could you please review the attached 
new patch [^JobManager.java.patch2]?



You can use this property to disable logging hopcount to database only for jobs 
with hopcount mode "keep unreachable documents, forever" specified in the "Hop 
Filters" tab.

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>Reporter: Mingchun Zhao
>Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1747:
--
Attachment: (was: JobManager.java.patch)

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Priority: Major
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723328#comment-17723328
 ] 

Mingchun Zhao commented on CONNECTORS-1747:
---

Hello,

I changed spec for new property as below. Could you please review the attached 
new patch [^JobManager.java.patch2]?



You can use this property to disable logging hopcount to database only for jobs 
with hopcount mode "keep unreachable documents, forever" specified in the "Hop 
Filters" tab.

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>Reporter: Mingchun Zhao
>Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722975#comment-17722975
 ] 

Mingchun Zhao commented on CONNECTORS-1747:
---

Hello, 
If there are no objections to the above patch, would it be okay to commit it in 
a couple of days?

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722789#comment-17722789
 ] 

Mingchun Zhao commented on CONNECTORS-1747:
---

Hello,

I’ve tried to create a patch for adding the below property to disable logging 
hopcount to the database as below.

"org.apache.manifoldcf.db.postgres.crewler.jobs.store_hopcount"

If you do not require hopcount from within, this will disable logging hotcount 
to the related database tables. This can increase throughput and reduce the 
rate of growth of the database. defaults to true(logging hopcount to the 
database).

In my testing with the attached patch, I compared the execution time of the 
same job with the property “store_hopcount” set to true and false. As a result, 
the throughput doubled and the rate of growth of the database was cut by more 
than half, while the number of crawled documents remained the same.

[^JobManager.java.patch]

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1747:
--
Attachment: JobManager.java.patch

> Add a property to disable logging hop count to database
> ---
>
> Key: CONNECTORS-1747
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
> Project: ManifoldCF
>  Issue Type: Improvement
>    Reporter: Mingchun Zhao
>Priority: Major
> Attachments: JobManager.java.patch
>
>
> If we do not require “Hop Filters“ feature, we need to consider to disable 
> logging records related to hopcount to database like "intrinsiclink" and 
> "hopcount" tables. This can increase throughput and reduce the rate of growth 
> of the database.
> I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (CONNECTORS-1747) Add a property to disable logging hop count to database

[Mingchun Zhao (Jira)]

Mingchun Zhao created CONNECTORS-1747:
-

 Summary: Add a property to disable logging hop count to database
 Key: CONNECTORS-1747
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1747
 Project: ManifoldCF
  Issue Type: Improvement
Reporter: Mingchun Zhao


If we do not require “Hop Filters“ feature, we need to consider to disable 
logging records related to hopcount to database like "intrinsiclink" and 
"hopcount" tables. This can increase throughput and reduce the rate of growth 
of the database.
I will try to create a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722207#comment-17722207
 ] 

Mingchun Zhao edited comment on CONNECTORS-1746 at 5/12/23 10:34 PM:
-

Hello,

Here is a patch for adding options for PostgreSQL’s “ANALYZE” command.
I’ve tried to add two properties to handle 'ANALYZE' command as below.

1. "org.apache.manifoldcf.db.postgres.analyzeatstart"
If this property is set to true, then analyze a table which is specified by 
property "org.apache.manifoldcf.db.postgres.analyze." at the start 
of job. defaults to false (not to run "ANALYZE" at the start).

2. "org.apache.manifoldcf.db.postgres.analyzeratethreshold"
If this property is set to a positive integer, then analyze a table which is 
specified by property "org.apache.manifoldcf.db.postgres.analyze." 
only when events per second drops below the threshold. defaults to 0(not to 
check event rate).

I tested using the attached patch and confirmed that the “ANALYZE” command was 
executed correctly in the above two situations. Especially, when MCF's 
throughput (event counts per second) dropped due to PostgreSQL's bad query 
plan, an “ANALYZE” command was executed and the MCF's performance recovered.

[^DBInterfacePostgreSQL.java.patch]


was (Author: mingchun.zhao):
Hello,

Here is a patch for adding options for PostgreSQL’s “ANALYZE” command.
I’ve tried to add two properties to handle 'ANALYZE' command as below.

1. "org.apache.manifoldcf.db.postgres.analyzeatstart"
If this property is set to true, then analyze a table which is specified by 
property "org.apache.manifoldcf.db.postgres.analyze." at the start 
of job. defaults to false (not to run "ANALYZE" at the start).

2. "org.apache.manifoldcf.db.postgres.analyzeratethreshold"
If this property is set to a positive integer, then analyze a table which is 
specified by property "org.apache.manifoldcf.db.postgres.analyze." 
only when events per second drops below the threshold. defaults to 1 (1 event 
processed per second).

I tested using the attached patch and confirmed that the “ANALYZE” command was 
executed correctly in the above two situations. Especially, when MCF's 
throughput (event counts per second) dropped due to PostgreSQL's bad query 
plan, an “ANALYZE” command was executed and the MCF's performance recovered.

[^DBInterfacePostgreSQL.java.patch]

> Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling 
> become extremely slow.
> --
>
> Key: CONNECTORS-1746
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Web connector
> Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
> database. 
>Reporter: Mingchun Zhao
>Assignee: Karl Wright
>Priority: Major
> Attachments: DBInterfacePostgreSQL.java.patch
>
>
> Sometimes, the crawling does not process any documents for a while and there 
> is nothing logged about long-running queries. The performance can be restored 
> by firing the 'ANALYZE' command manually. It seems that a bad query plan 
> caused this performance problem.
> Therefore, in addition to the current configuration parameter 
> 'org.apache.manifoldcf.db.postgres.analyze.', it is considered 
> necessary to execute the 'ANALYZE' even in the following situations.
> 1. When the number of records in the table exceeds the number required for 
> creating a execution plan after the job starts.
> 2. When the crawling performance slows down. For example, if the processing 
> rate of documents drops below a specified threshold.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722207#comment-17722207
 ] 

Mingchun Zhao edited comment on CONNECTORS-1746 at 5/12/23 3:25 PM:


Hello,

Here is a patch for adding options for PostgreSQL’s “ANALYZE” command.
I’ve tried to add two properties to handle 'ANALYZE' command as below.

1. "org.apache.manifoldcf.db.postgres.analyzeatstart"
If this property is set to true, then analyze a table which is specified by 
property "org.apache.manifoldcf.db.postgres.analyze." at the start 
of job. defaults to false (not to run "ANALYZE" at the start).

2. "org.apache.manifoldcf.db.postgres.analyzeratethreshold"
If this property is set to a positive integer, then analyze a table which is 
specified by property "org.apache.manifoldcf.db.postgres.analyze." 
only when events per second drops below the threshold. defaults to 1 (1 event 
processed per second).

I tested using the attached patch and confirmed that the “ANALYZE” command was 
executed correctly in the above two situations. Especially, when MCF's 
throughput (event counts per second) dropped due to PostgreSQL's bad query 
plan, an “ANALYZE” command was executed and the MCF's performance recovered.

[^DBInterfacePostgreSQL.java.patch]


was (Author: mingchun.zhao):
Hello,

Here is a patch for adding options for PostgreSQL’s “ANALYZE” command.
I’ve tried to add two properties to handle 'ANALYZE' command as below.
 # "org.apache.manifoldcf.db.postgres.analyzeatstart"
If this property is set to true, then analyze a table which is specified by 
property "org.apache.manifoldcf.db.postgres.analyze." at the start 
of job. defaults to false (not to run "ANALYZE" at the start).

 # "org.apache.manifoldcf.db.postgres.analyzeratethreshold"
If this property is set to a positive integer, then analyze a table which is 
specified by property "org.apache.manifoldcf.db.postgres.analyze." 
only when events per second drops below the threshold. defaults to 1 (1 event 
processed per second).

I tested using the attached patch and confirmed that the “ANALYZE” command was 
executed correctly in the above two situations. Especially, when MCF's 
throughput (event counts per second) dropped due to PostgreSQL's bad query 
plan, an “ANALYZE” command was executed and the MCF's performance recovered.

[^DBInterfacePostgreSQL.java.patch]

> Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling 
> become extremely slow.
> --
>
> Key: CONNECTORS-1746
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Web connector
> Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
> database. 
>Reporter: Mingchun Zhao
>Priority: Major
> Attachments: DBInterfacePostgreSQL.java.patch
>
>
> Sometimes, the crawling does not process any documents for a while and there 
> is nothing logged about long-running queries. The performance can be restored 
> by firing the 'ANALYZE' command manually. It seems that a bad query plan 
> caused this performance problem.
> Therefore, in addition to the current configuration parameter 
> 'org.apache.manifoldcf.db.postgres.analyze.', it is considered 
> necessary to execute the 'ANALYZE' even in the following situations.
> 1. When the number of records in the table exceeds the number required for 
> creating a execution plan after the job starts.
> 2. When the crawling performance slows down. For example, if the processing 
> rate of documents drops below a specified threshold.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722207#comment-17722207
 ] 

Mingchun Zhao commented on CONNECTORS-1746:
---

Hello,

Here is a patch for adding options for PostgreSQL’s “ANALYZE” command.
I’ve tried to add two properties to handle 'ANALYZE' command as below.
 # "org.apache.manifoldcf.db.postgres.analyzeatstart"
If this property is set to true, then analyze a table which is specified by 
property "org.apache.manifoldcf.db.postgres.analyze." at the start 
of job. defaults to false (not to run "ANALYZE" at the start).

 # "org.apache.manifoldcf.db.postgres.analyzeratethreshold"
If this property is set to a positive integer, then analyze a table which is 
specified by property "org.apache.manifoldcf.db.postgres.analyze." 
only when events per second drops below the threshold. defaults to 1 (1 event 
processed per second).

I tested using the attached patch and confirmed that the “ANALYZE” command was 
executed correctly in the above two situations. Especially, when MCF's 
throughput (event counts per second) dropped due to PostgreSQL's bad query 
plan, an “ANALYZE” command was executed and the MCF's performance recovered.

[^DBInterfacePostgreSQL.java.patch]

> Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling 
> become extremely slow.
> --
>
> Key: CONNECTORS-1746
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Web connector
> Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
> database. 
>Reporter: Mingchun Zhao
>Priority: Major
> Attachments: DBInterfacePostgreSQL.java.patch
>
>
> Sometimes, the crawling does not process any documents for a while and there 
> is nothing logged about long-running queries. The performance can be restored 
> by firing the 'ANALYZE' command manually. It seems that a bad query plan 
> caused this performance problem.
> Therefore, in addition to the current configuration parameter 
> 'org.apache.manifoldcf.db.postgres.analyze.', it is considered 
> necessary to execute the 'ANALYZE' even in the following situations.
> 1. When the number of records in the table exceeds the number required for 
> creating a execution plan after the job starts.
> 2. When the crawling performance slows down. For example, if the processing 
> rate of documents drops below a specified threshold.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1746:
--
Attachment: DBInterfacePostgreSQL.java.patch

> Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling 
> become extremely slow.
> --
>
> Key: CONNECTORS-1746
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Web connector
> Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
> database. 
>Reporter: Mingchun Zhao
>Priority: Major
> Attachments: DBInterfacePostgreSQL.java.patch
>
>
> Sometimes, the crawling does not process any documents for a while and there 
> is nothing logged about long-running queries. The performance can be restored 
> by firing the 'ANALYZE' command manually. It seems that a bad query plan 
> caused this performance problem.
> Therefore, in addition to the current configuration parameter 
> 'org.apache.manifoldcf.db.postgres.analyze.', it is considered 
> necessary to execute the 'ANALYZE' even in the following situations.
> 1. When the number of records in the table exceeds the number required for 
> creating a execution plan after the job starts.
> 2. When the crawling performance slows down. For example, if the processing 
> rate of documents drops below a specified threshold.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1746:
--
Description: 
Sometimes, the crawling does not process any documents for a while and there is 
nothing logged about long-running queries. The performance can be restored by 
firing the 'ANALYZE' command manually. It seems that a bad query plan caused 
this performance problem.

Therefore, in addition to the current configuration parameter 
'org.apache.manifoldcf.db.postgres.analyze.', it is considered 
necessary to execute the 'ANALYZE' even in the following situations.
1. When the number of records in the table exceeds the number required for 
creating a execution plan after the job starts.
2. When the crawling performance slows down. For example, if the processing 
rate of documents drops below a specified threshold.

  was:
Sometimes, the crawling does not process any documents for a while and there is 
nothing logged about long-running queries. The performance can be restored by 
firing the 'ANALYZE' command manually. It seems that a bad query plan caused 
this performance problem.

Therefore, in addition to the current configuration parameter 
'org.apache.manifoldcf.db.postgres.analyze.', it is considered 
necessary to execute the 'ANALYZE' even in the following situations.
1. When the number of records in the table exceeds the number required for 
creating a execution plan after the job starts.
2. When the crawling performance slows down. For example, if the processing 
rate of documents drops below a specified threshold.

So, how about adding two parameters to handle the timing of 'ANALYZE' execution 
as below?
1.'org.apache.manifoldcf.db.postgres.analyze..minimumrowcount'
Specify how many records should be inserted before carrying out an 'ANALYZE' on 
the specified table as the first time.defaults to 100.
2.'org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate'
Specify the minimum number of documents processed per minute. If the processing 
rate of documents drops below this threshold, the 'ANALYZE' will be executed. 
defaults to 1.


> Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling 
> become extremely slow.
> --
>
> Key: CONNECTORS-1746
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Web connector
> Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
> database. 
>Reporter: Mingchun Zhao
>Priority: Major
>
> Sometimes, the crawling does not process any documents for a while and there 
> is nothing logged about long-running queries. The performance can be restored 
> by firing the 'ANALYZE' command manually. It seems that a bad query plan 
> caused this performance problem.
> Therefore, in addition to the current configuration parameter 
> 'org.apache.manifoldcf.db.postgres.analyze.', it is considered 
> necessary to execute the 'ANALYZE' even in the following situations.
> 1. When the number of records in the table exceeds the number required for 
> creating a execution plan after the job starts.
> 2. When the crawling performance slows down. For example, if the processing 
> rate of documents drops below a specified threshold.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: About disabling hopcount tracking

[Mingchun Zhao]

Hi Karl,

Thank you for taking time out of your busy schedule to reply.

> There is an option on the "hopcount" tab of your job to disable hopcount

You mean setting "Hop count mode" to "keep unreachable documents,
forever" in the "Hop Filters" tab?
Yes, I did it, however, it seems that the records were still inserted
into the "intrinsiclink" and "hopcount" tables. Is there a way to tell
MCF not to insert data into those tables because operations on it can
become a performance bottleneck when the tables bloat?

Regards,
Mingchun

2023年5月10日(水) 19:53 Karl Wright :
>
> There is an option on the "hopcount" tab of your job to disable hopcount
> tracking entirely.
> Karl
>
> On Tue, May 9, 2023 at 11:49 PM Mingchun Zhao 
> wrote:
>
> > Hi Karl,
> >
> > Could you please advise me on tracking hopcount.
> > I'm using ManifoldCF 2.24 with PostgreSQL 12.14 as the database for now.
> > In my case, I don't need to use the 'Hop Filters' feature so I'd like
> > to disable tracking hopcount and reduce the insert/update/delete load
> > on the 'intrinsiclink' and 'hopcount' tables. So I have two questions
> > about this.
> > First, is there an option to disable tracking hopcount?
> > Second, if I disable tracking hopcount , can it affect other crawling
> > processes?
> >
> > Thank you in advance.
> > Kind regards,
> > Mingchun
> >

About disabling hopcount tracking

[Mingchun Zhao]

Hi Karl,

Could you please advise me on tracking hopcount.
I'm using ManifoldCF 2.24 with PostgreSQL 12.14 as the database for now.
In my case, I don't need to use the 'Hop Filters' feature so I'd like
to disable tracking hopcount and reduce the insert/update/delete load
on the 'intrinsiclink' and 'hopcount' tables. So I have two questions
about this.
First, is there an option to disable tracking hopcount?
Second, if I disable tracking hopcount , can it affect other crawling processes?

Thank you in advance.
Kind regards,
Mingchun

[jira] [Updated] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1746:
--
Description: 
Sometimes, the crawling does not process any documents for a while and there is 
nothing logged about long-running queries. The performance can be restored by 
firing the 'ANALYZE' command manually. It seems that a bad query plan caused 
this performance problem.

Therefore, in addition to the current configuration parameter 
'org.apache.manifoldcf.db.postgres.analyze.', it is considered 
necessary to execute the 'ANALYZE' even in the following situations.
1. When the number of records in the table exceeds the number required for 
creating a execution plan after the job starts.
2. When the crawling performance slows down. For example, if the processing 
rate of documents drops below a specified threshold.

So, how about adding two parameters to handle the timing of 'ANALYZE' execution 
as below?
1.'org.apache.manifoldcf.db.postgres.analyze..minimumrowcount'
Specify how many records should be inserted before carrying out an 'ANALYZE' on 
the specified table as the first time.defaults to 100.
2.'org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate'
Specify the minimum number of documents processed per minute. If the processing 
rate of documents drops below this threshold, the 'ANALYZE' will be executed. 
defaults to 1.

  was:
Sometimes, the crawling does not process any documents for a while and there is 
nothing logged about long-running queries. The performance can be restored by 
firing the 'ANALYZE' command manually. It seems that a bad query plan caused 
this performance problem.

Therefore, in addition to the current configuration parameter 
org.apache.manifoldcf.db.postgres.analyze. , it is considered 
necessary to execute the 'ANALYZE' even in the following situations.
1. When the number of records in the table exceeds the number required for 
creating a execution plan after the job starts.
2. When the crawling performance slows down. For example, if the processing 
rate of documents drops below a specified threshold.

So, how about adding two parameters to handle the timing of 'ANALYZE' execution 
as below?
1. `org.apache.manifoldcf.db.postgres.analyze..minimumrowcount`
Specify how many records should be inserted before carrying out an 'ANALYZE' on 
the specified table as the first time.defaults to 100.
2.`org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate`
Specify the number of documents processed in the last minute. If the actual 
processing rate falls below this, the 'ANALYZE' will be executed. defaults to 1.


> Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling 
> become extremely slow.
> --
>
> Key: CONNECTORS-1746
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Web connector
> Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
> database. 
>Reporter: Mingchun Zhao
>Priority: Major
>
> Sometimes, the crawling does not process any documents for a while and there 
> is nothing logged about long-running queries. The performance can be restored 
> by firing the 'ANALYZE' command manually. It seems that a bad query plan 
> caused this performance problem.
> Therefore, in addition to the current configuration parameter 
> 'org.apache.manifoldcf.db.postgres.analyze.', it is considered 
> necessary to execute the 'ANALYZE' even in the following situations.
> 1. When the number of records in the table exceeds the number required for 
> creating a execution plan after the job starts.
> 2. When the crawling performance slows down. For example, if the processing 
> rate of documents drops below a specified threshold.
> So, how about adding two parameters to handle the timing of 'ANALYZE' 
> execution as below?
> 1.'org.apache.manifoldcf.db.postgres.analyze..minimumrowcount'
> Specify how many records should be inserted before carrying out an 'ANALYZE' 
> on the specified table as the first time.defaults to 100.
> 2.'org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate'
> Specify the minimum number of documents processed per minute. If the 
> processing rate of documents drops below this threshold, the 'ANALYZE' will 
> be executed. defaults to 1.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1746:
--
Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the database.   
(was: I am using ManifoldCF 2.24 with PostgreSQL 12.14 as the database. )

> Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling 
> become extremely slow.
> --
>
> Key: CONNECTORS-1746
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Web connector
> Environment: Using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
> database. 
>Reporter: Mingchun Zhao
>Priority: Major
>
> Sometimes, the crawling does not process any documents for a while and there 
> is nothing logged about long-running queries. The performance can be restored 
> by firing the 'ANALYZE' command manually. It seems that a bad query plan 
> caused this performance problem.
> Therefore, in addition to the current configuration parameter 
> org.apache.manifoldcf.db.postgres.analyze. , it is considered 
> necessary to execute the 'ANALYZE' even in the following situations.
> 1. When the number of records in the table exceeds the number required for 
> creating a execution plan after the job starts.
> 2. When the crawling performance slows down. For example, if the processing 
> rate of documents drops below a specified threshold.
> So, how about adding two parameters to handle the timing of 'ANALYZE' 
> execution as below?
> 1. `org.apache.manifoldcf.db.postgres.analyze..minimumrowcount`
> Specify how many records should be inserted before carrying out an 'ANALYZE' 
> on the specified table as the first time.defaults to 100.
> 2.`org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate`
> Specify the number of documents processed in the last minute. If the actual 
> processing rate falls below this, the 'ANALYZE' will be executed. defaults to 
> 1.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1746:
--
Description: 
Sometimes, the crawling does not process any documents for a while and there is 
nothing logged about long-running queries. The performance can be restored by 
firing the 'ANALYZE' command manually. It seems that a bad query plan caused 
this performance problem.

Therefore, in addition to the current configuration parameter 
org.apache.manifoldcf.db.postgres.analyze. , it is considered 
necessary to execute the 'ANALYZE' even in the following situations.
1. When the number of records in the table exceeds the number required for 
creating a execution plan after the job starts.
2. When the crawling performance slows down. For example, if the processing 
rate of documents drops below a specified threshold.

So, how about adding two parameters to handle the timing of 'ANALYZE' execution 
as below?
1. `org.apache.manifoldcf.db.postgres.analyze..minimumrowcount`
Specify how many records should be inserted before carrying out an 'ANALYZE' on 
the specified table as the first time.defaults to 100.
2.`org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate`
Specify the number of documents processed in the last minute. If the actual 
processing rate falls below this, the 'ANALYZE' will be executed. defaults to 1.

  was:
Sometimes, the crawling does not process any documents for a while and there is 
nothing logged about long-running queries. The performance can be restored by 
firing the 'ANALYZE' command manually. It seems that a bad query plan caused 
this performance problem.

Therefore, in addition to the current configuration parameter 
org.apache.manifoldcf.db.postgres.analyze. , it is considered 
necessary to execute the 'ANALYZE' even in the following situations.
1. When the number of records in the table exceeds the number required for 
creating an query plan after the job starts.
2. When the crawling performance slows down. For example, if the document 
processing rate drops below a specified threshold. 

How about adding two parameters to handle the timing of 'ANALYZE' execution as 
below?
1. `org.apache.manifoldcf.db.postgres.analyze..minimumrowcount`
Specify how many records should be accumulated before carrying out an 'ANALYZE' 
on the specified table as the first time.defaults to 100.
2.`org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate`
Specify the number of documents processed in the last minute. If the actual 
processing rate falls below this, the 'ANALYZE' will be carrying out. defaults 
to 1.


> Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling 
> become extremely slow.
> --
>
> Key: CONNECTORS-1746
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Web connector
> Environment: I am using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
> database. 
>Reporter: Mingchun Zhao
>Priority: Major
>
> Sometimes, the crawling does not process any documents for a while and there 
> is nothing logged about long-running queries. The performance can be restored 
> by firing the 'ANALYZE' command manually. It seems that a bad query plan 
> caused this performance problem.
> Therefore, in addition to the current configuration parameter 
> org.apache.manifoldcf.db.postgres.analyze. , it is considered 
> necessary to execute the 'ANALYZE' even in the following situations.
> 1. When the number of records in the table exceeds the number required for 
> creating a execution plan after the job starts.
> 2. When the crawling performance slows down. For example, if the processing 
> rate of documents drops below a specified threshold.
> So, how about adding two parameters to handle the timing of 'ANALYZE' 
> execution as below?
> 1. `org.apache.manifoldcf.db.postgres.analyze..minimumrowcount`
> Specify how many records should be inserted before carrying out an 'ANALYZE' 
> on the specified table as the first time.defaults to 100.
> 2.`org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate`
> Specify the number of documents processed in the last minute. If the actual 
> processing rate falls below this, the 'ANALYZE' will be executed. defaults to 
> 1.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (CONNECTORS-1746) Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-1746:
--
Summary: Adding conditions to execute PostgreSQL's ANALYZE command to avoid 
crawling become extremely slow.  (was: Adding execution conditions of 
PostgreSQL's ANALYZE command to avoid crawling become extremely slow.)

> Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling 
> become extremely slow.
> --
>
> Key: CONNECTORS-1746
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
> Project: ManifoldCF
>  Issue Type: Improvement
>  Components: Web connector
> Environment: I am using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
> database. 
>Reporter: Mingchun Zhao
>Priority: Major
>
> Sometimes, the crawling does not process any documents for a while and there 
> is nothing logged about long-running queries. The performance can be restored 
> by firing the 'ANALYZE' command manually. It seems that a bad query plan 
> caused this performance problem.
> Therefore, in addition to the current configuration parameter 
> org.apache.manifoldcf.db.postgres.analyze. , it is considered 
> necessary to execute the 'ANALYZE' even in the following situations.
> 1. When the number of records in the table exceeds the number required for 
> creating an query plan after the job starts.
> 2. When the crawling performance slows down. For example, if the document 
> processing rate drops below a specified threshold. 
> How about adding two parameters to handle the timing of 'ANALYZE' execution 
> as below?
> 1. `org.apache.manifoldcf.db.postgres.analyze..minimumrowcount`
> Specify how many records should be accumulated before carrying out an 
> 'ANALYZE' on the specified table as the first time.defaults to 100.
> 2.`org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate`
> Specify the number of documents processed in the last minute. If the actual 
> processing rate falls below this, the 'ANALYZE' will be carrying out. 
> defaults to 1.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (CONNECTORS-1746) Adding execution conditions of PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

[Mingchun Zhao (Jira)]

Mingchun Zhao created CONNECTORS-1746:
-

 Summary: Adding execution conditions of PostgreSQL's ANALYZE 
command to avoid crawling become extremely slow.
 Key: CONNECTORS-1746
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1746
 Project: ManifoldCF
  Issue Type: Improvement
  Components: Web connector
 Environment: I am using ManifoldCF 2.24 with PostgreSQL 12.14 as the 
database. 
Reporter: Mingchun Zhao


Sometimes, the crawling does not process any documents for a while and there is 
nothing logged about long-running queries. The performance can be restored by 
firing the 'ANALYZE' command manually. It seems that a bad query plan caused 
this performance problem.

Therefore, in addition to the current configuration parameter 
org.apache.manifoldcf.db.postgres.analyze. , it is considered 
necessary to execute the 'ANALYZE' even in the following situations.
1. When the number of records in the table exceeds the number required for 
creating an query plan after the job starts.
2. When the crawling performance slows down. For example, if the document 
processing rate drops below a specified threshold. 

How about adding two parameters to handle the timing of 'ANALYZE' execution as 
below?
1. `org.apache.manifoldcf.db.postgres.analyze..minimumrowcount`
Specify how many records should be accumulated before carrying out an 'ANALYZE' 
on the specified table as the first time.defaults to 100.
2.`org.apache.manifoldcf.db.postgres.analyze..minimumprocessrate`
Specify the number of documents processed in the last minute. If the actual 
processing rate falls below this, the 'ANALYZE' will be carrying out. defaults 
to 1.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (CONNECTORS-1188) Missing Japanese image in end_user_documentation

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562871#comment-14562871
 ] 

Mingchun Zhao commented on CONNECTORS-1188:
---

Committed r1682238(dev_1x).
pull up changes r1682232 to the dev_1x branch from trunk.

 Missing Japanese image in end_user_documentation
 

 Key: CONNECTORS-1188
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1188
 Project: ManifoldCF
  Issue Type: Bug
  Components: Documentation
Affects Versions: ManifoldCF 1.9, ManifoldCF 2.1
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.10, ManifoldCF 2.2


 The image images/ja_JP/ad-configure-cache_ja_JP.PNG is missing.  This is 
 preventing us from building release candidates for 1.9 and 2.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-1188) Missing Japanese image in end_user_documentation

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562829#comment-14562829
 ] 

Mingchun Zhao commented on CONNECTORS-1188:
---

Committed r1682232(trunk).
added an image images/ja_JP/ad-configure-cache_ja_JP.PNG.
test: build the project part of the site.
$ cd site
$ echo $FORREST_HOME
/home/mzhao/apache-forrest-0.9
$ echo $JAVA_TOOL_OPTIONS
-Dfile.encoding=UTF8
$ forrest site
OK.



 Missing Japanese image in end_user_documentation
 

 Key: CONNECTORS-1188
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1188
 Project: ManifoldCF
  Issue Type: Bug
  Components: Documentation
Affects Versions: ManifoldCF 1.9, ManifoldCF 2.1
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.10, ManifoldCF 2.2


 The image images/ja_JP/ad-configure-cache_ja_JP.PNG is missing.  This is 
 preventing us from building release candidates for 1.9 and 2.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562928#comment-14562928
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

committed r1682252(trunk).
patch: end-user-documentation.xml.20150527.patch


 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.10, ManifoldCF 2.2

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.20150515.patch, 
 end-user-documentation.xml.20150527.patch, 
 end-user-documentation.xml.ActiveDirectory.patch, 
 end-user-documentation.xml.patch, end-user-documentation.xml.wikirepo.patch, 
 end-user-documentation.xml_IBM_FIlenet_P8.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562935#comment-14562935
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

committed r1682254(dev_1x).
pull up r1682252 to dev_1x from trunk.

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.10, ManifoldCF 2.2

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.20150515.patch, 
 end-user-documentation.xml.20150527.patch, 
 end-user-documentation.xml.ActiveDirectory.patch, 
 end-user-documentation.xml.patch, end-user-documentation.xml.wikirepo.patch, 
 end-user-documentation.xml_IBM_FIlenet_P8.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CONNECTORS-1188) Missing Japanese image in end_user_documentation

[Mingchun Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao resolved CONNECTORS-1188.
---
Resolution: Fixed

 Missing Japanese image in end_user_documentation
 

 Key: CONNECTORS-1188
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1188
 Project: ManifoldCF
  Issue Type: Bug
  Components: Documentation
Affects Versions: ManifoldCF 1.9, ManifoldCF 2.1
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.10, ManifoldCF 2.2


 The image images/ja_JP/ad-configure-cache_ja_JP.PNG is missing.  This is 
 preventing us from building release candidates for 1.9 and 2.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547081#comment-14547081
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Committed r1679826 into trunk.

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.10, ManifoldCF 2.2

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.20150515.patch, 
 end-user-documentation.xml.ActiveDirectory.patch, 
 end-user-documentation.xml.patch, end-user-documentation.xml.wikirepo.patch, 
 end-user-documentation.xml_IBM_FIlenet_P8.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547082#comment-14547082
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

pulls up commit r1679826 to dev_1x branch

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.10, ManifoldCF 2.2

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.20150515.patch, 
 end-user-documentation.xml.ActiveDirectory.patch, 
 end-user-documentation.xml.patch, end-user-documentation.xml.wikirepo.patch, 
 end-user-documentation.xml_IBM_FIlenet_P8.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-1188) Missing Japanese image in end_user_documentation

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497920#comment-14497920
 ] 

Mingchun Zhao commented on CONNECTORS-1188:
---

Karl, sorry, I noticed this just now. I'll deal with this by the next release.

 Missing Japanese image in end_user_documentation
 

 Key: CONNECTORS-1188
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1188
 Project: ManifoldCF
  Issue Type: Bug
  Components: Documentation
Affects Versions: ManifoldCF 1.9, ManifoldCF 2.1
Reporter: Karl Wright
Assignee: Mingchun Zhao
Priority: Blocker
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1


 The image images/ja_JP/ad-configure-cache_ja_JP.PNG is missing.  This is 
 preventing us from building release candidates for 1.9 and 2.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379780#comment-14379780
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

r1669100(trunk)

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.ActiveDirectory.patch, 
 end-user-documentation.xml.patch, 
 end-user-documentation.xml_IBM_FIlenet_P8.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379781#comment-14379781
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

r1669101(dev_1x)

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.ActiveDirectory.patch, 
 end-user-documentation.xml.patch, 
 end-user-documentation.xml_IBM_FIlenet_P8.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379779#comment-14379779
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Koizumi, thanks!

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.ActiveDirectory.patch, 
 end-user-documentation.xml.patch, 
 end-user-documentation.xml_IBM_FIlenet_P8.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache ManifoldCF Plugin for Solr 4.x version 2.1, RC0

[Mingchun Zhao]

Hi there,

I've got the same warning when testing for
apache-manifoldcf-solr-3.x-plugin-2.1 as Shinichiro met before.
The followings are the test results with java1.7 and java1.6.

===with java 1.7
$ uname -a
Darwin MingchunZhaos-MacBook-Air.local 13.4.0 Darwin Kernel Version
13.4.0: Wed Dec 17 19:05:52 PST 2014;
root:xnu-2422.115.10~1/RELEASE_X86_64 x86_64

$ pwd
/Users/mingchunzhao/mcf/apache-manifoldcf-solr-3.x-plugin-2.1

$ mvn --version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9;
2014-02-15T02:37:52+09:00)
Maven home: /Users/mingchunzhao/dev/apache-maven-3.2.1
Java version: 1.7.0_55, vendor: Oracle Corporation

$ mvn package assembly:assembly
... ...
---
 T E S T S
---
Running org.apache.solr.mcf.ManifoldCFQParserPluginTest
SLF4J: Failed to load class org.slf4j.impl.StaticLoggerBinder.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
further details.
WARNING: test class left thread running:
Thread[MultiThreadedHttpConnectionManager cleanup,5,main]
RESOURCE LEAK: test class left 1 thread(s) running
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.678
sec - in org.apache.solr.mcf.ManifoldCFQParserPluginTest
... ...


===with java 1.6
$ pwd
/Users/mingchunzhao/mcf/apache-manifoldcf-solr-3.x-plugin-2.1

$ mvn --version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9;
2014-02-15T02:37:52+09:00)
Maven home: /Users/mingchunzhao/dev/apache-maven-3.2.1
Java version: 1.6.0_65, vendor: Apple Inc.

$ mvn package assembly:assembly
... ...
---
 T E S T S
---
Running org.apache.solr.mcf.ManifoldCFQParserPluginTest
SLF4J: Failed to load class org.slf4j.impl.StaticLoggerBinder.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
further details.
WARNING: test class left thread running: Thread[AWT-AppKit,5,main]
WARNING: test class left thread running: Thread[AWT-Shutdown,5,main]
WARNING: test class left thread running:
Thread[MultiThreadedHttpConnectionManager cleanup,5,main]
RESOURCE LEAK: test class left 3 thread(s) running
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.887
sec - in org.apache.solr.mcf.ManifoldCFQParserPluginTest
Running org.apache.solr.mcf.ManifoldCFSCLoadTest
AWT blocker activation interrupted:
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at sun.awt.AWTAutoShutdown.activateBlockerThread(AWTAutoShutdown.java:315)
at sun.awt.AWTAutoShutdown.setToolkitBusy(AWTAutoShutdown.java:232)
at sun.awt.AWTAutoShutdown.notifyToolkitThreadBusy(AWTAutoShutdown.java:118)
Query time (milliseconds) = 12448
WARNING: test class left thread running: Thread[AWT-Shutdown,5,main]
RESOURCE LEAK: test class left 1 thread(s) running
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.446
sec - in org.apache.solr.mcf.ManifoldCFSCLoadTest
... ...


Regards,
Mingchun

2015-03-13 13:03 GMT+09:00 Shinichiro Abe shinichiro.ab...@gmail.com:
 Hi,

 +1 from me.

 checked cpmmit logs.
 executed mvn package assembly:assembly.

 However, I got a mvn test warning for 3.x plugin.
 Mingchun will report soon.

 Thanks,
 Shinichiro Abe

 On 2015/03/10, at 2:04, Karl Wright daddy...@gmail.com wrote:

 Please vote on whether to release the Apache ManifoldCF Plugin for Solr 4.x
 2.1, RC0.

 This release changes the build to use Maven rather than Ant, and also fixes
 a problem with multi-user-domain queries.

 The artifact can be found at:

 https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-solr-4.x-plugin-2.1

 There is also a tag at:

 https://svn.apache.org/repos/asf/manifoldcf/integration/solr-4.x/tags/release-2.1-RC0

 Thanks,
 Karl

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14332067#comment-14332067
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

r1661455(dev_1x)

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.patch, 
 end-user-documentation.xml_IBM_FIlenet_P8.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326753#comment-14326753
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Koizumi, thanks! got it.

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.patch, 
 end-user-documentation.xml_IBM_FIlenet_P8.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CONNECTORS-1164) Two view buttons in Japanese List of Authority Groups page

[Mingchun Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao resolved CONNECTORS-1164.
---
Resolution: Fixed

 Two view buttons in Japanese List of Authority Groups page
 

 Key: CONNECTORS-1164
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1164
 Project: ManifoldCF
  Issue Type: Bug
  Components: Framework core
Affects Versions: ManifoldCF 2.0.1
Reporter: KOIZUMI Satoru
Assignee: Mingchun Zhao
Priority: Minor
 Attachments: common_ja_JP.properties.patch


 There are two view(表示) button in List of Authority Groups(権限グループ一覧) page of 
 Japanese version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311290#comment-14311290
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Committed r1658155 into trunk.
Regular Expression User Mapping Connection　section(document and screenshots)

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304600#comment-14304600
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Koizumi, thanks! got it.

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml, 
 end-user-documentation.xml.patch


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CONNECTORS-1152) Update Japanese document of Technical Resources

[Mingchun Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao reassigned CONNECTORS-1152:
-

Assignee: Mingchun Zhao

 Update Japanese document of Technical Resources
 -

 Key: CONNECTORS-1152
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1152
 Project: ManifoldCF
  Issue Type: Improvement
  Components: Documentation
Affects Versions: ManifoldCF 2.0
Reporter: KOIZUMI Satoru
Assignee: Mingchun Zhao
Priority: Minor
 Attachments: technical-resources.xml.patch


 Japanese Technical Resources(開発者リソース) needs to be updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295078#comment-14295078
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

'ant doc' test passed.
Committed r1655264.

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-1152) Update Japanese document of Technical Resources

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295049#comment-14295049
 ] 

Mingchun Zhao commented on CONNECTORS-1152:
---

Committed r1655261 to trunk.

 Update Japanese document of Technical Resources
 -

 Key: CONNECTORS-1152
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1152
 Project: ManifoldCF
  Issue Type: Improvement
  Components: Documentation
Affects Versions: ManifoldCF 2.0
Reporter: KOIZUMI Satoru
Assignee: Mingchun Zhao
Priority: Minor
 Attachments: technical-resources.xml.patch


 Japanese Technical Resources(開発者リソース) needs to be updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296119#comment-14296119
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Karl, thanks!

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao updated CONNECTORS-853:
-
Attachment: CONNECTORS-853.patch

patch based on changes from Koizumi.

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: CONNECTORS-853.patch, end-user-documentation.xml


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-697) Need japanese translations for GoogleDrive connector

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286692#comment-14286692
 ] 

Mingchun Zhao commented on CONNECTORS-697:
--

For management, It would be better to create issues for documentations 
according to its own type, 
e.g.
- end user manual
- menu(or button) for connector xxx
- menu(or button) for job
and so on.
 

 Need japanese translations for GoogleDrive connector
 

 Key: CONNECTORS-697
 URL: https://issues.apache.org/jira/browse/CONNECTORS-697
 Project: ManifoldCF
  Issue Type: Bug
  Components: GoogleDrive connector
Affects Versions: ManifoldCF 1.3
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1


 Need japanese translations for GoogleDrive connector.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-697) Need japanese translations for GoogleDrive connector

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286722#comment-14286722
 ] 

Mingchun Zhao commented on CONNECTORS-697:
--

Karl, thanks. got it.

 Need japanese translations for GoogleDrive connector
 

 Key: CONNECTORS-697
 URL: https://issues.apache.org/jira/browse/CONNECTORS-697
 Project: ManifoldCF
  Issue Type: Bug
  Components: GoogleDrive connector
Affects Versions: ManifoldCF 1.3
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1


 Need japanese translations for GoogleDrive connector.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-1140) improve Japanese End User Manual

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14278309#comment-14278309
 ] 

Mingchun Zhao commented on CONNECTORS-1140:
---

Koizumi, thank you for your nice patch.
Karl, thank you putting this into trunk,dev_1x(I noticed this just now).

 improve Japanese End User Manual
 

 Key: CONNECTORS-1140
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1140
 Project: ManifoldCF
  Issue Type: Improvement
  Components: Documentation
Reporter: KOIZUMI Satoru
Assignee: Karl Wright
Priority: Minor
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1

 Attachments: end-user-documentation.xml.patch


 Thank you for your Japanese translation.
 I have made a patch to make it more fluent.
 I will explain each change if you need.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272502#comment-14272502
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Committed r1650742.
added the authority groups feature into Japanese end-user-documentation with 
related screen shots.


 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF next


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272516#comment-14272516
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Committed r1650747.
added the Amazon Cloud Search output connector feature into Japanese 
end-user-documentation with related screen shots.


 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272499#comment-14272499
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Committed r1650741.
added the transformation connection feature into Japanese 
end-user-documentation including related screen shots, and updated 
corresponding menus in Japanese Framework UI.

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF next


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272524#comment-14272524
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

TODO:

#Transformation Connection Types:
allowed documents
metadata adjuster
null transformer
tika extractor

#User Mapping Connection types
regexp mapper




 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF 1.9, ManifoldCF 2.1


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-853) Japanese end-user-documentation needs to be updated

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230771#comment-14230771
 ] 

Mingchun Zhao commented on CONNECTORS-853:
--

Hi Karl,
Got it, Let me look into this.

 Japanese end-user-documentation needs to be updated
 ---

 Key: CONNECTORS-853
 URL: https://issues.apache.org/jira/browse/CONNECTORS-853
 Project: ManifoldCF
  Issue Type: Task
  Components: Documentation
Affects Versions: ManifoldCF 1.5
Reporter: Karl Wright
Assignee: Mingchun Zhao
 Fix For: ManifoldCF next


 Japanese end-user-documentation needs to be updated to cover the new features 
 of 1.5.  Some new screen shots are needed too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CONNECTORS-1079) the parsing in TikaExtractor always return empty result

[Mingchun Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/CONNECTORS-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184445#comment-14184445
 ] 

Mingchun Zhao commented on CONNECTORS-1079:
---

Karl, thanks!
It does work pretty well.

 the parsing in TikaExtractor always return empty result
 ---

 Key: CONNECTORS-1079
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1079
 Project: ManifoldCF
  Issue Type: Bug
  Components: Tika extractor
Affects Versions: ManifoldCF 2.0
Reporter: Mingchun Zhao
Assignee: Karl Wright
 Fix For: ManifoldCF 1.8, ManifoldCF 2.0


 When I use latest trunk source(2.0) to try the Tika content extractor,It did 
 not return any expected results.
 I looked at it using debugging tools, found that the parser of Tika content 
 extractor does not return any data.
 I've tried to move lib/tika-core-1.6.jar into connector-lib/, 
 Then, the Tika content extractor returned data as expected.
 My configurations are as below:
 ==
 Transformation:
  Type: Tika content extractor
 Output:
  Type:Solr(Use extract update handler=false)
 Repository:
  type: Web
 Job:
  1.type: repository
  2.type: transformation
  3.type: output
 ==
 Maybe, it is related to CONNECTORS-1074(?), 
 It looks like that the place of tika-core-1.6.jar affects the result of 
 TikaExtractor.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CONNECTORS-1084) Missing resource 'WebcrawlerConnector.MatchMustHaveARegexpValue' in bundle 'org.apache.manifoldcf.crawler.connectors.webcrawler.common'

[Mingchun Zhao (JIRA)]

Mingchun Zhao created CONNECTORS-1084:
-

 Summary: Missing resource 
'WebcrawlerConnector.MatchMustHaveARegexpValue' in bundle 
'org.apache.manifoldcf.crawler.connectors.webcrawler.common'
 Key: CONNECTORS-1084
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1084
 Project: ManifoldCF
  Issue Type: Bug
  Components: Web connector
Affects Versions: ManifoldCF 2.0
Reporter: Mingchun Zhao
Assignee: Mingchun Zhao
Priority: Minor


An error occurred in web connector as below:

ERROR 2014-10-24 09:30:19,537 (qtp876209191-368) - Missing resource 
'WebcrawlerConnector.MatchMustHaveARegexpValue' in bundle 
'org.apache.manifoldcf.crawler.connectors.webcrawler.common' for locale 'ja'
java.util.MissingResourceException: Can't find resource for bundle 
java.util.PropertyResourceBundle, key 
WebcrawlerConnector.MatchMustHaveARegexpValue
at java.util.ResourceBundle.getObject(ResourceBundle.java:395)
at java.util.ResourceBundle.getString(ResourceBundle.java:355)
... ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CONNECTORS-1084) Missing resource 'WebcrawlerConnector.MatchMustHaveARegexpValue' in bundle 'org.apache.manifoldcf.crawler.connectors.webcrawler.common'

[Mingchun Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CONNECTORS-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingchun Zhao resolved CONNECTORS-1084.
---
   Resolution: Fixed
Fix Version/s: ManifoldCF 2.0

Committed r1634202(trunk).

 Missing resource 'WebcrawlerConnector.MatchMustHaveARegexpValue' in bundle 
 'org.apache.manifoldcf.crawler.connectors.webcrawler.common'
 ---

 Key: CONNECTORS-1084
 URL: https://issues.apache.org/jira/browse/CONNECTORS-1084
 Project: ManifoldCF
  Issue Type: Bug
  Components: Web connector
Affects Versions: ManifoldCF 2.0
Reporter: Mingchun Zhao
Assignee: Mingchun Zhao
Priority: Minor
 Fix For: ManifoldCF 2.0


 An error occurred in web connector as below:
 ERROR 2014-10-24 09:30:19,537 (qtp876209191-368) - Missing resource 
 'WebcrawlerConnector.MatchMustHaveARegexpValue' in bundle 
 'org.apache.manifoldcf.crawler.connectors.webcrawler.common' for locale 'ja'
 java.util.MissingResourceException: Can't find resource for bundle 
 java.util.PropertyResourceBundle, key 
 WebcrawlerConnector.MatchMustHaveARegexpValue
 at java.util.ResourceBundle.getObject(ResourceBundle.java:395)
 at java.util.ResourceBundle.getString(ResourceBundle.java:355)
 ... ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)