Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi There, In order to resolve junit test failure with Solr 9.x Output connector, I modified MockSolrService to support HTTP2C. I prepared a PR for this: https://github.com/apache/manifoldcf/pull/157 I confirmed that all test cases of ant test passed. ``` test: BUILD SUCCESSFUL Total time: 51

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

That is indeed reassuring. It probably means that the test needs some changes, is all. But we cannot be sure the newer Zookeeper wouldn't mess things up without having the test be successful at least with the older zookeeper. Karl On Fri, Oct 27, 2023 at 7:32 AM Guylaine BASSETTE <

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Sorry for not executing all of the tests: I'm quite new to the MCF project, and I focused only on the core-framework tests as it was the key aspect highlighted in the related jira issue. I didn't know I had to run other tests such as the IT ones. It is to be noted that we tested the MCF with

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Okay, well I wouldnt have approved the upgrade had I known that the tests didn't pass! So we need to understand the problem as soon as possible. Karl On Fri, Oct 27, 2023 at 2:50 AM Guylaine BASSETTE < guylaine.basse...@francelabs.com> wrote: > Hello all, > > Sadly, the error remains the same

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hello all, Sadly, the error remains the same even with the previous Zookeeper version (3.8.0). Actually, I'm not able to pass the test since we have passed to Solr 9. Le 27/10/2023 à 01:51, Karl Wright a écrit : It is possible that Solr needs the older version of Zookeeper. If you swap out

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

It is possible that Solr needs the older version of Zookeeper. If you swap out the current one and replace it with the one the version of SolrJ we use references, does the test pass then? If it does, we're going to have to figure out how to address the fact that we have two connectors that each

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

I ran the ant test using Guylaine's patch and the Kafka IT test passed! However, the Solr IT test failure still occurs. ``` run-IT-HSQLDB: [junit] Testsuite: org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT [junit] Configuration file successfully read [junit] [main]

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi Mingchun, If you want, you can access my repository directly: https://github.com/guylaine-bassette/manifoldcf I have made a Fork of Github ManifoldCF. Use the branch: Fix-kafka-IT-tests. It is up to date with the last trunk commit. Le 26/10/2023 à 17:13, Mingchun Zhao a écrit : Sorry,

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Sorry, there was a typo: track -> trunk 2023年10月27日(金) 0:12 Mingchun Zhao : > Hi Guylaine, Thanks! > Was your patch based on the latest track? Unfortunately, I was unable to > apply your patch in my environment. > > 2023年10月26日(木) 20:44 Guylaine BASSETTE : > >> Hello Mingchun, >> >> As mentioned

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi Guylaine, Thanks! Was your patch based on the latest track? Unfortunately, I was unable to apply your patch in my environment. 2023年10月26日(木) 20:44 Guylaine BASSETTE : > Hello Mingchun, > > As mentioned in my previous e-mail to Karl, my patch contained other files > to be updated. I don't

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Oh yes sorry, when I made the patch, I put a bad version of the build.xml. But I see you didn't take the other files I have modified... Is it because they was bad according to you? Le 25/10/2023 à 22:42, Karl Wright a écrit : I see you have committed this. You missed a few spots; the extra

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi there, Allow me to share my `ant test` result using the latest Github trunk. I got a lot of `Broker may not be available` warnings in the Kafka IT test and it eventually failed. ``` ... [junit] [kafka-producer-network-thread | producer-3] INFO org.apache.kafka.clients.NetworkClient -

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

I see you have committed this. You missed a few spots; the extra jar was mentioned in multiple places. I committed another fix to correct that. Karl On Wed, Oct 25, 2023 at 10:46 AM Guylaine BASSETTE < guylaine.basse...@francelabs.com> wrote: > Hello Karl, > > Thank you very much for this

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

I was able to reproduce the problem last night. I believe the cause may well be that we've moved too many dependencies to the framework level. Specifically, I think perhaps only zookeeper and netty should run there, but the Scala library probably needs to run at the same classloader level as the

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Try doing svn update and deleting your test-materials directory contents. Then ant download-dependencies. You don't get a link error after that when you do run-IT-HSQLDB . Karl On Tue, Oct 24, 2023 at 9:10 AM Guylaine BASSETTE < guylaine.basse...@francelabs.com> wrote: > Hi Karl and Mingchun,

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi Karl and Mingchun, Thanks again ! My bad for the Zookeeper dependencies. Actually I've made a mistake using in my IDE dependencies analyzer. Regarding Kafka tests, a light search brings me to the "spark-streaming-kafka" dependency that might be missing... At the mean time, I continue

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

I missed a place - connector-build.xml. Updated now. Now we don't get a link exception, but neither does the kafka test work. It seems to be unable to start zookeeper even though all the dependencies are now there. Will need to look at this after work. Karl On Mon, Oct 23, 2023 at 11:32 PM

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Thanks. I've tried `ant test` with the latest trunk. As a result, the kafka test failed as below. ``` [junit] Testcase: sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): Caused an ERROR [junit] io/netty/handler/ssl/SslContext [junit]

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Okay, I updated zookeeper properly in build.xml and framework/build.xml, with the two new dependencies, and the zookeeper tests pass. I haven't tried the kafka or solr tests yet. Karl On Mon, Oct 23, 2023 at 9:29 PM Karl Wright wrote: > Unless I know what kafka is using zookeeper for, this

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Unless I know what kafka is using zookeeper for, this would seem risky to me. Zookeeper is meant to coordinate processes; it may not work for one process to be using different versions of zookeeper than the others. It looks like the original change to kafka you reverted had the proper

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

I reverted zookeeper version to 3.8.0 to avoid linkage error on the multiThreadZooKeeperLockTest: [junit] Caused by: java.lang.ClassNotFoundException: io.netty.handler.ssl.SslContext I've prepared a PR here: https://github.com/apache/manifoldcf/pull/156 Just a heads up, `ant test` still hangs on

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

The dependencies would be in the zookeeper pom. Maven would follow them automatically which is why it is insufficient to assume that if maven works so will ant. You can use mvn dependency:tree to find what Maven is actually pulling in. Karl On Mon, Oct 23, 2023 at 11:04 AM Guylaine BASSETTE <

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

I launch these tests with a Maven build and everything is OK. Yet no netty dependencies are requiered... I don't understand were this SslContext is called... Le 23/10/2023 à 16:25, Karl Wright a écrit : Yes, that is indicating that zookeeper is looking for a specific netty class that it isn't

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Yes, that is indicating that zookeeper is looking for a specific netty class that it isn't finding. That is why I think there is now a zookeeper dependency we aren't including in the classpaths that include zookeeper. Karl On Mon, Oct 23, 2023 at 10:23 AM Mingchun Zhao wrote: > Karl, Thanks!

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Karl, Thanks! I think I reproduced that multiThreadZooKeeperLockTest error when I running `ant test`, will look into this. ``` [junit] - --- [junit] Testcase: multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks):

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

This is all you need to do: ant clean-core-deps ant make-core-deps ant clean ant test Karl On Mon, Oct 23, 2023 at 9:55 AM Mingchun Zhao wrote: > Hi Guylaine, Thanks! > > > Thanks for all your sharing, it's very helpful! I'll continue... > > I'll look into it some more too. If I have any

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi Guylaine, Thanks! > Thanks for all your sharing, it's very helpful! I'll continue... I'll look into it some more too. If I have any other information I'll share it with you. 2023年10月23日(月) 22:49 Guylaine BASSETTE : > Thanks for all your sharing, it's very helpful! I'll continue... > >

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Thanks for all your sharing, it's very helpful! I'll continue... Sorry, French and English mixed up! Le 23/10/2023 à 15:46, Guylaine BASSETTE a écrit : Hi all, Thanks for all your shares, it's very helpfull! Merci pour tous vos partages, c'est très utile ! Je poursuis... Le 23/10/2023 à

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi all, Thanks for all your shares, it's very helpfull! Merci pour tous vos partages, c'est très utile ! Je poursuis... Le 23/10/2023 à 15:31, Karl Wright a écrit : I can't give advice on the test; this is something FranceLabs should look at. However, nothing of what you are doing will

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

I can't give advice on the test; this is something FranceLabs should look at. However, nothing of what you are doing will affect the zookeeper tests in framework. That's a totally different issue. Karl On Mon, Oct 23, 2023 at 9:20 AM Mingchun Zhao wrote: > Hi Karl and Guylaine, > > > I hope

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi Karl and Guylaine, > I hope and think it's just a problem specific to the test. Missing updates or incompatible dependencies... Allow me to share with you what I'm working on. I've tried to support http2C within the Solr output connector junit test, but got another unhandled solr exception

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

> Then, wherever zookeeper is mentioned in framework/build.xml, a reference to those dependencies must also be included. It looks like zookeeper*.jar was already included in connector-test-classpath within kafka/build.xml. ```

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi, That just downloads zookeeper. But apparently the zookeeper version required by Kafka now has dependencies of its own. Otherwise the zookeeper tests wouldn't fail with linkage errors. The dependencies need to be identified and added in several places. The first place is to the

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi Karl, > Mingchun, did you add the jar(s) that the new zookeeper needs to the build.xml download section? Are the following settings correct? Or do you have an old version of zookeeper-*.jar left in your environment? ```build.xml

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Well, that was interesting. Didn't get very far, because the dependency updates committed broke Zookeeper again: [junit] Testcase: multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks): Caused an ERROR [junit] io/netty/handler/ssl/SslContext [junit]

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

I begin to suspect that the problem may be human error. If you don't do "ant clean-core-deps; ant make-core-deps", but instead just use "ant make-core-deps", you could have incompatible versions of several libraries in your classpath for the tests. I'll try today to verify whether that might be

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi Karl and Mingchun, Thanks for your work on the last few issues. I join you on this Solr testing problem. That said, we've tested this new connector in our application, with a FileShare job and everything was OK. I hope and think it's just a problem specific to the test. Missing updates

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi Karl, Thanks! > so I wonder if, once again, there's a problem with dependencies for the version of Solr they chose. I'll take a look at this issue. 2023年10月20日(金) 9:50 Karl Wright : > This connector FranceLabs updated. The problem seems to occur at a basic > level during http2

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

This connector FranceLabs updated. The problem seems to occur at a basic level during http2 communication, so I wonder if, once again, there's a problem with dependencies for the version of Solr they chose. Karl On Thu, Oct 19, 2023 at 8:32 PM Mingchun Zhao wrote: > About the test

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

About the test "SolrCrawlHSQLDBIT" failure, it seems that "IO exception during indexing http://test58.txt: frame_size_error/invalid_frame_length" error is occurring on the ManifoldCF side. - command: ``` manifoldcf/connectors/solr% ant run-IT-HSQLDB run-IT-HSQLDB: [junit] Testsuite:

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

I'm having trouble with the Solr test `ant run-IT-HSQLDB`. It's been over an hour since I ran the test and it still doesn't finish. It seems that the problem is probably due to the Manifold job not finishing. Do you have any ideas or advice? Thanks! ``` ~/ManifoldCF/manifoldcf/connectors/solr%

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi Karl, I've tried to update Kafka and its dependencies to the latest version including zookeeper, and confirmed tha kafka test run-IT-HSQLDB passed as below: ``` ~manifoldcf% cd connectors/kafka ~manifoldcf/connectors/kafka/% ant run-IT-HSQLDB BUILD SUCCESSFUL Total time: 1 minute 19 seconds

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Hi, It looks like the latest release of Kafka is 3.6.0. I'd try setting that version in the pom for connectors/kafka and doing mvn install. Then you can see what dependencies it wants by: mvn dependency:tree It may be that Kafka no longer even requires zookeeper - I didn't find it in a cursory

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Kalr, Thanks! Though I'm not familiar with kafka, I'll try to find out what's causing the error as much as possible. Kind Regards, Mingchun 2023年10月14日(土) 14:07 Karl Wright : > Yes, this seems to be something related to zookeeper update and the Kafka > library version we're using. > > Someone

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

Yes, this seems to be something related to zookeeper update and the Kafka library version we're using. Someone will need to dig into what is going wrong here before we can release. I don't know how widely used the kafka connector is but if it is lightly used we can perhaps not distribute the

Re: [CANCEL][VOTE] Release ManifoldCF 2.26, RC0

By applying r1912939, I was able to confirm that the kafka test compile error has disappeared when running `ant test`. Thanks, Karl! However, the following error occurred on subsequent test runs. ``` [junit] Testsuite: org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT [junit]

[CANCEL][VOTE] Release ManifoldCF 2.26, RC0

r1912939 fixes this but I need to spin a new RC. Karl On Fri, Oct 13, 2023 at 8:46 AM Karl Wright wrote: > Yes I get the same thing; a test needs to be updated. > > [javac] >

Re: [VOTE] Release ManifoldCF 2.26, RC0

Yes I get the same thing; a test needs to be updated. [javac] C:\wip\mcf\release-2.26-branch\connectors\kafka\connector\src\test\java\org\apache\manifoldcf\agents\output\kafka\ZooKeeperLocal.java:45: error: unreported exception AdminServerException; must be caught or declared to be thrown

Re: [VOTE] Release ManifoldCF 2.26, RC0

There was a Zookeeper dependency change this release. I wonder if there is a test that needs to be updated. Let me try and see. Karl On Fri, Oct 13, 2023 at 4:51 AM Piergiorgio Lucidi wrote: > Hi Mingchun, > > thank you for your message and I was trying to build ManifoldCF using > OpenJDK

Re: [VOTE] Release ManifoldCF 2.26, RC0

Hi Mingchun, thank you for your message and I was trying to build ManifoldCF using OpenJDK 17 so probably in the future for supporting this version of Java we should include Jaxb libraries as well. The build is ok now and I can compile and package everything correctly. Unfortunately executing

Re: [VOTE] Release ManifoldCF 2.26, RC0

Hi Piergiorgio, FYI, Allow me to share the java and ant versions and the build steps I ran in my environment. ``` $ java --version openjdk 11.0.11 2021-04-20 OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9) OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9,

Re: [VOTE] Release ManifoldCF 2.26, RC0

Hi folks, it seems that I can't compile the CswsConnector: [javac] public List getAttributeGroups() [javac] ^ [javac] symbol: class AttributeGroup [javac] location: class CswsConnector.ObjectInformation [javac]

Re: [VOTE] Release ManifoldCF 2.26, RC0

Hi, +1 from France Labs Regards, Guylaine France Labs – Your knowledge, now Datafari Enterprise Search – Découvrez la version 5 / Discover our version 5 www.datafari.com Le 11/10/2023 à 23:12, Furkan KAMACI a écrit : Hi, +1! Kind regards, Furkan Kamaci On Wed, Oct

Re: [VOTE] Release ManifoldCF 2.26, RC0

Hi, +1! Kind regards, Furkan Kamaci On Wed, Oct 11, 2023 at 11:13 PM Mingchun Zhao wrote: > Hi, > Ran all the tests. > +1 from me. > > Kind regards, > Mingchun > > 2023年10月12日(木) 4:29 Cihad Guzel : > > > Hi, > > > > +1 from me > > > > Regards, > > Cihad Guzel > > > > 11 Eki 2023 Çar, saat

Re: [VOTE] Release ManifoldCF 2.26, RC0

Hi, Ran all the tests. +1 from me. Kind regards, Mingchun 2023年10月12日(木) 4:29 Cihad Guzel : > Hi, > > +1 from me > > Regards, > Cihad Guzel > > 11 Eki 2023 Çar, saat 15:08 tarihinde Karl Wright > şunu > yazdı: > > > Please vote on whether to release Apache ManifoldCF 2.26, RC0. > > This

Re: [VOTE] Release ManifoldCF 2.26, RC0

Hi, +1 from me Regards, Cihad Guzel 11 Eki 2023 Çar, saat 15:08 tarihinde Karl Wright şunu yazdı: > Please vote on whether to release Apache ManifoldCF 2.26, RC0. > This release is the first release that requires at least Java 11, and it > also includes a new CSV connector along with support

[VOTE] Release ManifoldCF 2.26, RC0

Please vote on whether to release Apache ManifoldCF 2.26, RC0. This release is the first release that requires at least Java 11, and it also includes a new CSV connector along with support for Solr 9. The release artifact can be found at: