Hi there, Allow me to share my `ant test` result using the latest Github trunk. I got a lot of `Broker may not be available` warnings in the Kafka IT test and it eventually failed.
``` ... [junit] [kafka-producer-network-thread | producer-3] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Node 0 disconnected. [junit] [kafka-producer-network-thread | producer-3] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Connection to node 0 (/192.168.10.103:9092) could not be established. Broker may not be available. [junit] [kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 0 disconnected. [junit] [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 0 (/192.168.10.103:9092) could not be established. Broker may not be available. [junit] [kafka-producer-network-thread | producer-2] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Node 0 disconnected. [junit] [kafka-producer-network-thread | producer-2] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Connection to node 0 (/192.168.10.103:9092) could not be established. Broker may not be available. [junit] [kafka-producer-network-thread | producer-3] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Node 0 disconnected. [junit] [kafka-producer-network-thread | producer-3] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Connection to node 0 (/192.168.10.103:9092) could not be established. Broker may not be available. [junit] ------------- ---------------- --------------- [junit] Testcase: sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): Caused an ERROR [junit] ManifoldCF did not terminate in the allotted time of 120000 milliseconds [junit] org.apache.manifoldcf.core.interfaces.ManifoldCFException: ManifoldCF did not terminate in the allotted time of 120000 milliseconds [junit] at org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT.waitJobInactive(APISanityHSQLDBIT.java:289) [junit] at org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT.sanityCheck(APISanityHSQLDBIT.java:177) [junit] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit] [junit] BUILD FAILED /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following error occurred while executing this line: /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following error occurred while executing this line: /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1104: Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed Total time: 38 minutes 13 seconds ``` 2023年10月26日(木) 5:42 Karl Wright <daddy...@gmail.com>: > I see you have committed this. You missed a few spots; the extra jar was > mentioned in multiple places. I committed another fix to correct that. > > Karl > > > On Wed, Oct 25, 2023 at 10:46 AM Guylaine BASSETTE < > guylaine.basse...@francelabs.com> wrote: > > > Hello Karl, > > > > Thank you very much for this update! I have tested your suggestions and > > kafka ITs tests ended successfully. :-) > > > > Here you can find the patch. > > > > My only doubt is this warning I had in some parts of the kafka test: > > > > ``` > > > > [junit] [Controller-0-to-broker-0-send-thread] INFO > > org.apache.kafka.clients.NetworkClient - [Controller id=0, > > targetBrokerId=0] Node 0 disconnected. > > [junit] [Controller-0-to-broker-0-send-thread] WARN > > org.apache.kafka.clients.NetworkClient - [Controller id=0, > > targetBrokerId=0] Connection to node 0 (guylaine-virtual-machine/ > > 127.0.1.1:9092) could not be established. Broker may not be available. > > [junit] [Controller-0-to-broker-0-send-thread] WARN > > kafka.controller.RequestSendThread - [RequestSendThread controllerId=0] > > Controller 0's connection to broker guylaine-virtual-machine:9092 (id: 0 > > rack: null) was unsuccessful > > [junit] java.io.IOException: Connection to > > guylaine-virtual-machine:9092 (id: 0 rack: null) failed. > > [junit] at > > > org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70) > > [junit] at > > > kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:298) > > [junit] at > > > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:251) > > [junit] at > > > org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130) > > [junit] [Controller-0-to-broker-0-send-thread] INFO > > org.apache.kafka.clients.NetworkClient - [Controller id=0, > > targetBrokerId=0] Client requested connection close from node 0 > > [junit] [controller-event-thread] INFO state.change.logger - > > [Controller id=0 epoch=2] Sending LeaderAndIsr request to broker 0 with 1 > > become-leader and 0 become-follower partitions > > ``` > > > > I have also ran an "ant test". It run core-framework and ITs tests until > > mongoDB connector with this fail: > > ``` > > [junit] Testcase: > > > sanityCheck(org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT): > > Caused an ERROR > > [junit] Could not start process: <EOF> > > [junit] java.lang.RuntimeException: Could not start process: <EOF> > > [junit] at > > > de.flapdoodle.embed.mongo.AbstractMongoProcess.onAfterProcessStart(AbstractMongoProcess.java:81) > > [junit] at > > > de.flapdoodle.embed.process.runtime.AbstractProcess.<init>(AbstractProcess.java:115) > > [junit] at > > > de.flapdoodle.embed.mongo.AbstractMongoProcess.<init>(AbstractMongoProcess.java:54) > > [junit] at > > de.flapdoodle.embed.mongo.MongodProcess.<init>(MongodProcess.java:50) > > [junit] at > > > de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:44) > > [junit] at > > > de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:34) > > [junit] at > > de.flapdoodle.embed.process.runtime.Executable.start(Executable.java:109) > > [junit] at > > > org.apache.manifoldcf.agents.output.mongodboutput.tests.BaseITHSQLDB.setUpMongoDB(BaseITHSQLDB.java:72) > > [junit] at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > [junit] at > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] > > [junit] > > [junit] Testcase: > > > sanityCheck(org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT): > > Caused an ERROR > > [junit] null > > [junit] java.lang.NullPointerException > > [junit] at > > > org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT.removeTestArea(APISanityHSQLDBIT.java:109) > > [junit] at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > [junit] at > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > ``` > > > > And no more success for the ITs tests of Solr connector... > > > > Guylaine > > Le 25/10/2023 à 12:52, Karl Wright a écrit : > > > > I was able to reproduce the problem last night. I believe the cause may > > well be that we've moved too many dependencies to the framework level. > > Specifically, I think perhaps only zookeeper and netty should run there, > > but the Scala library probably needs to run at the same classloader level > > as the other Scala jars, so it should be moved back into connectors/kafka > > and removed from framework/build.xml and from build.xml and from > > connector-build.xml. > > > > Sadly I'm completely snowed under until the weekend so it will need to > wait > > until then unless someone else wants to try this. > > > > Karl > > > > > > On Tue, Oct 24, 2023 at 9:12 AM Karl Wright <daddy...@gmail.com> < > daddy...@gmail.com> wrote: > > > > > > Try doing svn update and deleting your test-materials directory contents. > > Then ant download-dependencies. You don't get a link error after that > when > > you do run-IT-HSQLDB . > > > > Karl > > > > On Tue, Oct 24, 2023 at 9:10 AM Guylaine BASSETTE < > guylaine.basse...@francelabs.com> wrote: > > > > > > Hi Karl and Mingchun, > > > > Thanks again ! > > > > My bad for the Zookeeper dependencies. Actually I've made a mistake > > using in my IDE dependencies analyzer. > > > > Regarding Kafka tests, a light search brings me to the > > "spark-streaming-kafka" dependency that might be missing... > > > > At the mean time, I continue my effort on Solr connector tests. > > > > > > For any use, here is the errors I get: > > > > ``` > > > > [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler - > > Stopped o.e.j.w.WebAppContext@7c28c1{ManifoldCF General API > > > > > Webapp,/mcf-api-service,null,STOPPED}{/home/guylaine/IdeaProjects/mon-manifoldcf/dist/web/war/mcf-api-service.war} > > [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler > > - Stopped o.e.j.w.WebAppContext@588ffeb{ManifoldCF Authorities API > > > > > Webapp,/mcf-authority-service,null,STOPPED}{/home/guylaine/IdeaProjects/mon-manifoldcf/dist/web/war/mcf-authority-service.war} > > [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler > > - Stopped o.e.j.w.WebAppContext@71a3a190{ManifoldCF Crawler > > > > > Interface,/mcf-crawler-ui,null,STOPPED}{/home/guylaine/IdeaProjects/mon-manifoldcf/dist/web/war/mcf-crawler-ui.war} > > [junit] ------------- ---------------- --------------- > > [junit] Testcase: > > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): > > Caused an ERROR > > [junit] 'scala.collection.immutable.ArraySeq > > scala.runtime.ScalaRunTime$.wrapRefArray(java.lang.Object[])' > > [junit] java.lang.NoSuchMethodError: > > 'scala.collection.immutable.ArraySeq > > scala.runtime.ScalaRunTime$.wrapRefArray(java.lang.Object[])' > > [junit] at > > kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala:338) > > [junit] at > > kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1603) > > [junit] at > > > > > org.apache.manifoldcf.agents.output.kafka.KafkaLocal.<init>(KafkaLocal.java:31) > > [junit] at > > > > > org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.setupKafka(BaseITHSQLDB.java:86) > > [junit] at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > [junit] at > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] > > [junit] > > [junit] Testcase: > > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): > > Caused an ERROR > > [junit] null > > [junit] java.lang.NullPointerException > > [junit] at > > > > > org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.cleanUpKafka(BaseITHSQLDB.java:92) > > [junit] at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > [junit] at > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > ``` > > > > Le 24/10/2023 à 12:30, Karl Wright a écrit : > > > > I missed a place - connector-build.xml. Updated now. > > > > Now we don't get a link exception, but neither does the kafka test work. > > It seems to be unable to start zookeeper even though all the > > > > dependencies > > > > are now there. Will need to look at this after work. > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 11:32 PM Mingchun Zhao< > > > > mingchun.zha...@gmail.com> > > > > wrote: > > > > > > Thanks. I've tried `ant test` with the latest trunk. As a result, the > > > > kafka > > > > test failed as below. > > ``` > > [junit] Testcase: > > > > > > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): > > > > Caused an ERROR > > [junit] io/netty/handler/ssl/SslContext > > [junit] java.lang.NoClassDefFoundError: > > > > io/netty/handler/ssl/SslContext > > > > [junit] at > > > > > > > > > org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106) > > > > [junit] at > > > > > > > > > org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96) > > > > [junit] at > > org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92) > > [junit] at > > org.apache.zookeeper.common.ZKConfig.<init>(ZKConfig.java:61) > > [junit] at > > > > > > org.apache.zookeeper.client.ZKClientConfig.<init>(ZKClientConfig.java:69) > > > > [junit] at > > > > kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1620) > > > > [junit] at > > > > kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1603) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.agents.output.kafka.KafkaLocal.<init>(KafkaLocal.java:31) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.setupKafka(BaseITHSQLDB.java:86) > > > > [junit] at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > [junit] at > > > > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > [junit] at > > > > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > [junit] Caused by: java.lang.ClassNotFoundException: > > io.netty.handler.ssl.SslContext > > [junit] at > > > > > > > > > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) > > > > [junit] at > > > > > > > > > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) > > > > [junit] at > > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) > > [junit] > > [junit] > > [junit] Testcase: > > > > > > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): > > > > Caused an ERROR > > [junit] null > > [junit] java.lang.NullPointerException > > [junit] at > > > > > > > > > org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.cleanUpKafka(BaseITHSQLDB.java:92) > > > > [junit] at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > [junit] at > > > > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > [junit] at > > > > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > [junit] > > [junit] > > > > BUILD FAILED > > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following > > error occurred while executing this line: > > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following > > error occurred while executing this line: > > > > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102: > > > > Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed > > ``` > > > > 2023年10月24日(火) 11:00 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com > >: > > > > > > Okay, I updated zookeeper properly in build.xml and > > > > framework/build.xml, > > > > with the two new dependencies, and the zookeeper tests pass. I > > > > haven't > > > > tried the kafka or solr tests yet. > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 9:29 PM Karl Wright<daddy...@gmail.com> < > daddy...@gmail.com> > > > > wrote: > > > > Unless I know what kafka is using zookeeper for, this would seem > > > > risky > > > > to > > > > me. Zookeeper is meant to coordinate processes; it may not work for > > > > one > > > > process to be using different versions of zookeeper than the others. > > > > It looks like the original change to kafka you reverted had the > > > > proper > > > > dependencies but they absolutely needed to be included in the right > > classpaths and they weren't - they were only included in the kafka > > > > tests. > > > > I will look at this perhaps at the latest this weekend, but I won't > > > > commit > > > > this patch. > > > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 5:14 PM Mingchun Zhao < > > > > mingchun.zha...@gmail.com > > > > wrote: > > > > > > I reverted zookeeper version to 3.8.0 to avoid linkage error on the > > multiThreadZooKeeperLockTest: > > [junit] Caused by: java.lang.ClassNotFoundException: > > io.netty.handler.ssl.SslContext > > > > I've prepared a PR here:https://github.com/apache/manifoldcf/pull/156 > > > > Just a heads up, `ant test` still hangs on the Solr Output connector > > > > test: > > > > ``` > > run-IT-HSQLDB: > > [junit] Testsuite: > > org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT > > [junit] Configuration file successfully read > > [junit] [main] INFO org.eclipse.jetty.util.log - Logging > > > > initialized > > > > @7027ms to org.eclipse.jetty.util.log.Slf4jLog > > [junit] [main] INFO org.eclipse.jetty.server.Server - > > jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: > > 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9 > > [junit] [main] INFO org.eclipse.jetty.server.session - > > DefaultSessionIdManager workerName=node0 > > [junit] [main] INFO org.eclipse.jetty.server.session - No > > SessionScavenger set, using defaults > > [junit] [main] INFO org.eclipse.jetty.server.session - node0 > > Scavenging > > every 660000ms > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler > > > > - > > > > Started o.e.j.w.WebAppContext@1517f633{ManifoldCF Crawler > > > > > > > > > Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-7554899724821045499/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-crawler-ui.war} > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler > > > > - > > > > Started o.e.j.w.WebAppContext@4fe01803{ManifoldCF Authorities API > > > > > > > > > Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-7701836901953162228/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-authority-service.war} > > > > [junit] Creating mock service > > [junit] Mock service created > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler > > > > - > > > > Started o.e.j.w.WebAppContext@13d186db{ManifoldCF General API > > > > > > > > > Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-2609388202403972652/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-api-service.war} > > > > [junit] [main] INFO org.eclipse.jetty.server.AbstractConnector > > > > - > > > > Started ServerConnector@3bd55d8{HTTP/1.1, (http/1.1)}{0.0.0.0:8346} > > [junit] [main] INFO org.eclipse.jetty.server.Server - Started > > > > @9054ms > > > > [junit] [main] INFO org.eclipse.jetty.server.Server - > > jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: > > 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9 > > [junit] [main] INFO org.eclipse.jetty.server.session - > > DefaultSessionIdManager workerName=node0 > > [junit] [main] INFO org.eclipse.jetty.server.session - No > > SessionScavenger set, using defaults > > [junit] [main] INFO org.eclipse.jetty.server.session - node0 > > Scavenging > > every 600000ms > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler > > > > - > > > > Started o.e.j.s.ServletContextHandler@6f4ade6e > > > > {/solr,null,AVAILABLE} > > > > [junit] [main] INFO org.eclipse.jetty.server.AbstractConnector > > > > - > > > > Started ServerConnector@30e6a763{HTTP/1.1, (http/1.1)}{0.0.0.0:8188 > > > > } > > > > [junit] [main] INFO org.eclipse.jetty.server.Server - Started > > > > @9064ms > > > > [junit] [main] INFO org.eclipse.jetty.server.AbstractConnector > > > > - > > > > Stopped ServerConnector@30e6a763{HTTP/1.1, (http/1.1)}{0.0.0.0:8188 > > > > } > > > > [junit] [main] INFO org.eclipse.jetty.server.session - node0 > > > > Stopped > > > > scavenging > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler > > > > - > > > > Stopped o.e.j.s.ServletContextHandler@6f4ade6e{/solr,null,STOPPED} > > ``` > > > > 2023年10月24日(火) 1:40 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com > >: > > > > > > The dependencies would be in the zookeeper pom. Maven would follow > > > > them > > > > automatically which is why it is insufficient to assume that if > > > > maven > > > > works > > > > so will ant. > > > > You can use mvn dependency:tree to find what Maven is actually > > > > pulling > > > > in. > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 11:04 AM Guylaine BASSETTE < > guylaine.basse...@francelabs.com> wrote: > > > > > > I launch these tests with a Maven build and everything is OK. Yet > > > > no > > > > netty dependencies are requiered... I don't understand were this > > SslContext is called... > > > > Le 23/10/2023 à 16:25, Karl Wright a écrit : > > > > Yes, that is indicating that zookeeper is looking for a specific > > > > netty > > > > class that it isn't finding. That is why I think there is now a > > > > zookeeper > > > > dependency we aren't including in the classpaths that include > > > > zookeeper. > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 10:23 AM Mingchun Zhao< > > > > mingchun.zha...@gmail.com > > > > wrote: > > > > > > Karl, Thanks! > > I think I reproduced that multiThreadZooKeeperLockTest error > > > > when I > > > > running `ant test`, will look into this. > > ``` > > [junit] ------------- ---------------- --------------- > > [junit] Testcase: > > > > > > > > > multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks): > > > > Caused an ERROR > > [junit] io/netty/handler/ssl/SslContext > > [junit] java.lang.NoClassDefFoundError: > > > > io/netty/handler/ssl/SslContext > > > > [junit] at > > > > > > > > > org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106) > > > > [junit] at > > > > > > > > > org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96) > > > > [junit] at > > org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92) > > [junit] at > > org.apache.zookeeper.common.ZKConfig.<init>(ZKConfig.java:61) > > [junit] at > > > > > > org.apache.zookeeper.client.ZKClientConfig.<init>(ZKClientConfig.java:69) > > > > [junit] at > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:643) > > [junit] at > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:567) > > [junit] at > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:734) > > [junit] at > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:448) > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.<init>(ZooKeeperConnection.java:66) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnectionPool.grab(ZooKeeperConnectionPool.java:48) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.obtainGlobalReadLock(ZooKeeperLockObject.java:190) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.LockObject.enterReadLock(LockObject.java:310) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:271) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks.enterReadLock(TestZooKeeperLocks.java:125) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks$ReaderThread.run(TestZooKeeperLocks.java:204) > > > > [junit] Caused by: java.lang.ClassNotFoundException: > > io.netty.handler.ssl.SslContext > > [junit] at > > > > > > > > > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) > > > > [junit] at > > > > > > > > > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) > > > > [junit] at > > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) > > [junit] > > [junit] > > > > BUILD FAILED > > ``` > > > > 2023年10月23日(月) 23:02 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com > >: > > > > > > This is all you need to do: > > > > ant clean-core-deps > > ant make-core-deps > > ant clean > > ant test > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 9:55 AM Mingchun Zhao < > > > > mingchun.zha...@gmail.com > > > > wrote: > > > > > > Hi Guylaine, Thanks! > > > > > > Thanks for all your sharing, it's very helpful! I'll > > > > continue... > > > > I'll look into it some more too. If I have any other > > > > information > > > > I'll > > > > share > > > > it with you. > > > > 2023年10月23日(月) 22:49 Guylaine BASSETTE < > > > > guylaine.basse...@francelabs.com > > > > : > > > > > > Thanks for all your sharing, it's very helpful! I'll > > > > continue... > > > > Sorry, French and English mixed up! > > > > Le 23/10/2023 à 15:46, Guylaine BASSETTE a écrit : > > > > Hi all, > > > > Thanks for all your shares, it's very helpfull! Merci pour > > > > tous > > > > vos > > > > partages, c'est très utile ! Je poursuis... > > > > Le 23/10/2023 à 15:31, Karl Wright a écrit : > > > > I can't give advice on the test; this is something > > > > FranceLabs > > > > should > > > > look > > at. > > However, nothing of what you are doing will affect the > > > > zookeeper > > > > tests in > > framework. That's a totally different issue. > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 9:20 AM Mingchun Zhao< > > > > mingchun.zha...@gmail.com > > > > wrote: > > > > > > Hi Karl and Guylaine, > > > > > > I hope and think it's just a problem specific to the > > > > test. > > > > Missing > > > > updates or incompatible dependencies... > > > > Allow me to share with you what I'm working on. I've > > > > tried > > > > to > > > > support > > > > http2C within the Solr output connector junit test, but > > > > got > > > > another > > > > unhandled solr exception when I ran `ant run-IT-HSQLDB`. > > > > - source code change > > ``` > > diff --git > > > > > > > > > a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests > > > > /MockSolrService.java > > > > > > > > > b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output > > > > /solr/tests/MockSolrService.java > > index 237ade09c..3fb558f52 100644 > > --- > > > > > > > > > a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo > > > > lrService.java > > +++ > > > > > > > > > b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo > > > > lrService.java > > @@ -18,7 +18,10 @@ > > */ > > package > > > > org.apache.manifoldcf.agents.output.solr.tests; > > > > +import > > > > org.eclipse.jetty.http2.server.HTTP2CServerConnectionFactory; > > > > import org.eclipse.jetty.servlet.ServletHolder; > > +import org.eclipse.jetty.server.HttpConfiguration; > > +import org.eclipse.jetty.server.HttpConnectionFactory; > > import org.eclipse.jetty.server.Server; > > import org.eclipse.jetty.server.ServerConnector; > > import > > > > org.eclipse.jetty.servlet.ServletContextHandler; > > > > @@ -40,7 +43,10 @@ public class MockSolrService > > public MockSolrService() > > { > > server = new Server(new QueuedThreadPool(35)); > > - ServerConnector connector = new > > > > ServerConnector(server); > > > > + HttpConfiguration config = new HttpConfiguration(); > > + HttpConnectionFactory http1 = new > > > > HttpConnectionFactory(config); > > > > + HTTP2CServerConnectionFactory http2c = new > > HTTP2CServerConnectionFactory(config); > > + ServerConnector connector = new > > > > ServerConnector(server, > > > > http1, > > > > http2c); > > connector.setPort(8188); > > server.addConnector(connector); > > servlet = new SolrServlet(); > > @@ -111,6 +117,7 @@ public class MockSolrService > > res.getWriter().printf(Locale.ROOT, "<solr>\n"); > > res.getWriter().printf(Locale.ROOT, > > > > "</solr>\n"); > > > > res.getWriter().flush(); > > } > > ``` > > > > - run Junit test > > ``` > > ~manifoldcf/connectors/solr% ant run-IT-HSQLDB > > ``` > > > > - confirm test-HSQLDB-output/manifoldcf.log > > ``` > > ERROR 2023-10-23T22:10:29,902 (Worker thread '24') - > > > > Exception > > > > tossed: > > > > Unhandled Solr exception during indexinghttp://test70.txt > > > > (200): > > > > Error > > > > from server athttp://localhost:8188/solr: Expected mime > > > > type > > > > application/octet-stream but got application/xml. > > > > <result> > > > > <doc name="something"/> > > </result> > > > > > > > > org.apache.manifoldcf.core.interfaces.ManifoldCFException: > > > > Unhandled > > > > Solr > > exception during indexinghttp://test70.txt (200): Error > > > > from > > > > server > > > > at > > > > http://localhost:8188/solr: Expected mime type > > > > application/octet-stream > > > > but > > got application/xml. <result> > > <doc name="something"/> > > </result> > > > > at > > > > > > > > > org.apache.manifoldcf.agents.output.solr.HttpPoster.handleSolrException(HttpPoster.java:389) > > > > ~[classes/:?] > > at > > > > > > > > > org.apache.manifoldcf.agents.output.solr.HttpPoster.indexPost(HttpPoster.java:544) > > > > ~[classes/:?] > > at > > > > > > > > > org.apache.manifoldcf.agents.output.solr.SolrConnector.addOrReplaceDocumentWithException(SolrConnector.java:522) > > > > ~[classes/:?] > > at > > > > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3214) > > > > ~[mcf-agents.jar:?] > > at > > > > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$OutputAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3395) > > > > ~[mcf-agents.jar:?] > > at > > > > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3065) > > > > ~[mcf-agents.jar:?] > > at > > > > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2696) > > > > ~[mcf-agents.jar:?] > > at > > > > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:750) > > > > ~[mcf-agents.jar:?] > > at > > > > > > > > > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1585) > > > > ~[mcf-pull-agent.jar:?] > > at > > > > > > > > > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1550) > > > > ~[mcf-pull-agent.jar:?] > > at > > > > > > > > > org.apache.manifoldcf.crawler.tests.TestingRepositoryConnector.processDocuments(TestingRepositoryConnector.java:84) > > > > ~[mcf-pull-agent-tests.jar:?] > > at > > > > > > > > > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:402) > > > > ~[mcf-pull-agent.jar:?] > > ERROR 2023-10-23T22:10:29,902 (Worker thread '10') - > > > > Exception > > > > tossed: > > > > Unhandled Solr exception during indexinghttp://test72.txt > > > > (200): > > > > Error > > > > from server athttp://localhost:8188/solr: Expected mime > > > > type > > > > application/octet-stream but got application/xml. > > > > <result> > > > > <doc name="something"/> > > </result> > > ``` > > > > Could you give me some advice? > > > > 2023年10月23日(月) 22:01 Mingchun Zhao< > > > > mingchun.zha...@gmail.com > > > > : > > > > Then, wherever zookeeper is mentioned in > > > > framework/build.xml, > > > > a > > > > reference to those dependencies must also be included. > > > > It looks like zookeeper*.jar was already included in > > connector-test-classpath within kafka/build.xml. > > ``` > > <path id="connector-test-classpath"> > > <path > > > > refid="mcf-connector-build.connector-test-classpath"/> > > > > <fileset dir="../../lib"> > > <include name="zookeeper*.jar"/> > > <include name="kafka-clients*.jar"/> > > <include name="lz4*.jar"/> > > <include name="snappy-java*.jar"/> > > <include name="metrics-core*.jar"/> > > </fileset> > > ``` > > > > > > 2023年10月23日(月) 21:50 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com > >: > > > > > > Hi, > > > > That just downloads zookeeper. But apparently the > > > > zookeeper > > > > version > > > > required by Kafka now has dependencies of its own. > > > > Otherwise > > > > the > > > > zookeeper > > tests wouldn't fail with linkage errors. > > > > The dependencies need to be identified and added in > > > > several > > > > places. The > > first place is to the download-zookeeper part of the > > > > root > > > > build.xml > > > > script. Then, wherever zookeeper is mentioned in > > framework/build.xml, a > > reference to those dependencies must also be included. > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 8:32 AM Mingchun Zhao < > > > > mingchun.zha...@gmail.com> > > > > wrote: > > > > > > Hi Karl, > > > > > > Mingchun, did you add the jar(s) that the new > > > > zookeeper > > > > needs > > > > to > > > > the > > > > build.xml download section? > > > > Are the following settings correct? Or do you have an > > > > old > > > > version > > > > of > > > > zookeeper-*.jar left in your environment? > > > > ```build.xml > > <target name="download-zookeeper"> > > <mkdir dir="lib"/> > > <antcall target="download-via-maven"> > > <param name="target" value="lib"/> > > <param name="project-path" > > value="org/apache/zookeeper"/> > > <param name="artifact-version" > > > > value="${zookeeper.version}"/> > > > > <param name="artifact-name" > > > > value="zookeeper"/> > > > > <param name="artifact-type" > > > > value="jar"/> > > > > </antcall> > > ... ... > > ``` > > > > Regards, > > Mingchun > > > > > > 2023年10月23日(月) 21:19 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com > >: > > > > > > Well, that was interesting. > > > > Didn't get very far, because the dependency updates > > > > committed > > > > broke > > > > Zookeeper again: > > > > [junit] Testcase: > > > > > > > > > multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks): > > > > Caused an ERROR > > [junit] io/netty/handler/ssl/SslContext > > [junit] java.lang.NoClassDefFoundError: > > > > io/netty/handler/ssl/SslContext > > > > [junit] at > > > > > > > > > org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106) > > > > [junit] at > > > > > > > > > org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96) > > > > [junit] at > > > > > > org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92) > > > > [junit] at > > > > > > org.apache.zookeeper.common.ZKConfig.<init>(ZKConfig.java:61) > > > > [junit] at > > > > > > org.apache.zookeeper.client.ZKClientConfig.<init>(ZKClientConfig.java:69) > > > > [junit] at > > > > > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:643) > > > > [junit] at > > > > > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:567) > > > > [junit] at > > > > > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:734) > > > > [junit] at > > > > > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:448) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74) > > > > [junit] at > > > > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.<init>(ZooKeeperConnection.java:66) > > > > It looks like the even newer Zookeeper version has a > > > > newer > > > > dependency > > > > that > > > > isn't being included in the basic classpath, but > > > > should > > > > be. > > > > Mingchun, > > > > did > > > > you add the jar(s) that the new zookeeper needs to > > > > the > > > > build.xml > > > > download > > > > section? If so, can you remind me what they were? > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 8:11 AM Karl Wright< > > > > daddy...@gmail.com > > > > wrote: > > > > I begin to suspect that the problem may be human > > > > error. > > > > If you don't do "ant clean-core-deps; ant > > > > make-core-deps", > > > > but > > > > instead > > > > just use "ant make-core-deps", you could have > > > > incompatible > > > > versions > > > > of > > > > several libraries in your classpath for the tests. > > > > I'll > > > > try > > > > today > > > > to > > > > verify whether that might be happening by trying the > > > > tests > > > > locally > > > > myself. > > > > Karl > > > > > > On Mon, Oct 23, 2023 at 7:57 AM Guylaine BASSETTE < > guylaine.basse...@francelabs.com> wrote: > > > > > > Hi Karl and Mingchun, > > > > Thanks for your work on the last few issues. I join > > > > you on > > > > this > > > > Solr > > > > testing problem. > > > > That said, we've tested this new connector in our > > > > application, > > > > with a > > > > FileShare job and everything was OK. > > > > I hope and think it's just a problem specific to > > > > the > > > > test. > > > > Missing > > > > updates or incompatible dependencies... > > > > Le 20/10/2023 à 02:58, Mingchun Zhao a écrit : > > > > Hi Karl, Thanks! > > > > > > so I wonder if, once again, there's a problem > > > > with > > > > dependencies > > > > for > > > > the > > > > version of Solr they chose. > > > > I'll take a look at this issue. > > > > 2023年10月20日(金) 9:50 Karl Wright< > > > > daddy...@gmail.com > > > > : > > > > This connector FranceLabs updated. The problem > > > > seems to > > > > occur > > > > at a > > > > basic > > > > level during http2 communication, so I wonder if, > > > > once > > > > again, > > > > there's a > > > > problem with dependencies for the version of Solr > > > > they > > > > chose. > > > > Karl > > > > > > On Thu, Oct 19, 2023 at 8:32 PM Mingchun Zhao< > > > > mingchun.zha...@gmail.com> > > > > wrote: > > > > > > About the test "SolrCrawlHSQLDBIT" failure, it > > > > seems > > > > that > > > > "IO > > > > exception > > > > during indexinghttp://test58.txt: > > > > frame_size_error/invalid_frame_length" > > > > error is occurring on the ManifoldCF side. > > > > - command: > > ``` > > manifoldcf/connectors/solr% ant run-IT-HSQLDB > > > > run-IT-HSQLDB: > > [junit] Testsuite: > > > > > > org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT > > > > ... ... > > ``` > > > > - I checked > > > > "connectors/solr/test-HSQLDB-output/manifoldcf.log": > > > > ``` > > WARN 2023-10-20T09:14:56,635 (Worker thread > > > > '18') - > > > > IO > > > > exception > > > > during > > > > indexinghttp://test58.txt: > > > > frame_size_error/invalid_frame_length > > > > java.io.IOException: > > > > frame_size_error/invalid_frame_length > > > > at > > > > org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:566) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > org.eclipse.jetty.http2.HTTP2Session.access$2700(HTTP2Session.java:80) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:1857) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.HTTP2Session$StreamsState.access$400(HTTP2Session.java:1436) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:511) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:506) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onConnectionFailure(Parser.java:414) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.HTTP2Connection$ParserListener.onConnectionFailure(HTTP2Connection.java:397) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:223) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:215) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > org.eclipse.jetty.http2.parser.Parser.connectionFailure(Parser.java:209) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > org.eclipse.jetty.http2.parser.Parser.parseHeader(Parser.java:155) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:121) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:261) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:183) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:138) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:361) > > > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at org.eclipse.jetty.io > > > > .FillInterest.fillable(FillInterest.java:105) > > > > ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at org.eclipse.jetty.io > > > > .ChannelEndPoint$1.run(ChannelEndPoint.java:104) > > > > ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:69) > > > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:350) > > > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305) > > > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > > > at > > > > > > > > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:271) > > > > ~[solr-solrj-9.1.0.jar:9.1.0 > > > > aa4f3d98ab19c201e7f3c74cd14c99174148616d > > > > - > > > > ishan - 2022-11-11 13:00:47] > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > > > ~[?:?] > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > > > ~[?:?] > > at java.lang.Thread.run(Thread.java:829) ~[?:?] > > ``` > > > > 2023年10月19日(木) 20:37 Mingchun > > Zhao<mingchun.zha...@gmail.com > > > > : > > > > I'm having trouble with the Solr test `ant > > > > run-IT-HSQLDB`. > > > > It's been over an hour since I ran the test and > > > > it > > > > still > > > > doesn't > > > > finish. > > > > It seems that the problem is probably due to > > > > the > > > > Manifold > > > > job > > > > not > > > > finishing. > > Do you have any ideas or advice? Thanks! > > > > ``` > > ~/ManifoldCF/manifoldcf/connectors/solr% ant > > > > run-IT-HSQLDB > > > > Buildfile: > > > > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/connectors/solr/build.xml > > > > ... ... > > run-IT-HSQLDB: > > [junit] Testsuite: > > > > > > org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT > > > > [junit] Configuration file successfully > > > > read > > > > [junit] [main] INFO > > > > org.eclipse.jetty.util.log > > > > - > > > > Logging > > > > initialized > > > > @7416ms to org.eclipse.jetty.util.log.Slf4jLog > > [junit] [main] INFO > > > > org.eclipse.jetty.server.Server - > > > > jetty-9.4.48.v20220622; built: > > > > 2022-06-21T20:42:25.880Z; > > > > git: > > > > 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm > > > > 11.0.11+9 > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.session > > > > - > > > > DefaultSessionIdManager workerName=node0 > > [junit] [main] INFO > > > > org.eclipse.jetty.server.session > > > > - > > > > No > > > > SessionScavenger set, using defaults > > [junit] [main] INFO > > > > org.eclipse.jetty.server.session > > > > - > > > > node0 > > > > Scavenging every 600000ms > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler - > > > > Started o.e.j.w.WebAppContext@4bdc8b5d > > > > {ManifoldCF > > > > Crawler > > > > > Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-17189990303852051874/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-crawler-ui.war} > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler - > > > > Started o.e.j.w.WebAppContext@3bcd426c > > > > {ManifoldCF > > > > Authorities > > > > API > > > > > Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-1541416273384984853/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-authority-service.war} > > > > [junit] Creating mock service > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler - > > > > Started o.e.j.w.WebAppContext@5f14a673 > > > > {ManifoldCF > > > > General > > > > API > > > > > > > Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-12421572948842353797/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-api-service.war} > > > > [junit] Mock service created > > [junit] [main] INFO > > > > org.eclipse.jetty.server.AbstractConnector - > > > > Started ServerConnector@70325d20{HTTP/1.1, > > > > (http/1.1)}{ > > > > 0.0.0.0:8346} > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.Server - > > > > Started > > > > @9616ms > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.Server - > > > > jetty-9.4.48.v20220622; built: > > > > 2022-06-21T20:42:25.880Z; > > > > git: > > > > 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm > > > > 11.0.11+9 > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.session > > > > - > > > > DefaultSessionIdManager workerName=node0 > > [junit] [main] INFO > > > > org.eclipse.jetty.server.session > > > > - > > > > No > > > > SessionScavenger set, using defaults > > [junit] [main] INFO > > > > org.eclipse.jetty.server.session > > > > - > > > > node0 > > > > Scavenging every 600000ms > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler - > > > > Started o.e.j.s.ServletContextHandler@7808f638 > > > > {/solr,null,AVAILABLE} > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.AbstractConnector - > > > > Started ServerConnector@2ab5afc7{HTTP/1.1, > > > > (http/1.1)}{ > > > > 0.0.0.0:8188} > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.Server - > > > > Started > > > > @9626ms > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.AbstractConnector - > > > > Stopped ServerConnector@2ab5afc7{HTTP/1.1, > > > > (http/1.1)}{ > > > > 0.0.0.0:8188} > > > > [junit] [main] INFO > > > > org.eclipse.jetty.server.session > > > > - > > > > node0 > > > > Stopped > > > > scavenging > > [junit] [main] INFO > > > > org.eclipse.jetty.server.handler.ContextHandler - > > > > Stopped o.e.j.s.ServletContextHandler@7808f638 > > > > {/solr,null,STOPPED} > > > > ``` > > > > 2023年10月19日(木) 20:05 Mingchun Zhao< > > > > mingchun.zha...@gmail.com > > > > : > > > > Hi Karl, > > > > I've tried to update Kafka and its > > > > dependencies > > > > to > > > > the > > > > latest > > > > version > > > > including zookeeper, > > and confirmed tha kafka test run-IT-HSQLDB > > > > passed > > > > as > > > > below: > > > > ``` > > ~manifoldcf% cd connectors/kafka > > ~manifoldcf/connectors/kafka/% ant > > > > run-IT-HSQLDB > > > > BUILD SUCCESSFUL > > Total time: 1 minute 19 seconds > > ``` > > > > Also, I prepared a PR for this issue: > https://github.com/apache/manifoldcf/pull/155 > > > > 2023年10月19日(木) 7:12 Karl Wright< > > > > daddy...@gmail.com>: > > > > Hi, > > > > It looks like the latest release of Kafka is > > > > 3.6.0. > > > > I'd try setting that version in the pom for > > > > connectors/kafka > > > > and > > > > doing > > > > mvn > > install. Then you can see what dependencies > > > > it > > > > wants > > > > by: > > > > mvn dependency:tree > > > > It may be that Kafka no longer even requires > > > > zookeeper > > > > - > > > > I > > > > didn't > > > > find > > > > it > > > > in a cursory inspection. But the > > > > dependency:tree > > > > would > > > > be > > > > the > > > > final > > > > word. > > > > Karl > > > > > > On Sat, Oct 14, 2023 at 2:17 AM Mingchun > > > > Zhao < > > > > mingchun.zha...@gmail.com > > > > wrote: > > > > > > Kalr, Thanks! > > Though I'm not familiar with kafka, I'll try > > > > to > > > > find > > > > out > > > > what's > > > > causing the > > > > error as much as possible. > > > > Kind Regards, > > Mingchun > > > > > > 2023年10月14日(土) 14:07 Karl Wright< > > > > daddy...@gmail.com > > > > : > > > > Yes, this seems to be something related to > > > > zookeeper > > > > update > > > > and > > > > the > > > > Kafka > > > > library version we're using. > > > > Someone will need to dig into what is going > > > > wrong > > > > here > > > > before > > > > we > > > > can > > > > release. I don't know how widely used the > > > > kafka > > > > connector is > > > > but > > > > if > > > > it > > > > is > > > > lightly used we can perhaps not distribute > > > > the > > > > connector > > > > any > > > > longer. But > > > > that would be a last choice. > > > > Karl > > > > > > On Fri, Oct 13, 2023 at 12:12 PM Mingchun > > > > Zhao > > > > < > > > > mingchun.zha...@gmail.com> > > > > wrote: > > > > > > By applying r1912939, I was able to > > > > confirm > > > > that > > > > the > > > > kafka > > > > test > > > > compile > > > > error has disappeared when running `ant > > > > test`. > > > > Thanks, Karl! > > > > However, the following error occurred on > > > > subsequent > > > > test > > > > runs. > > > > ``` > > [junit] Testsuite: > > > > > > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT > > > > [junit] Tests run: 1, Failures: 0, Errors: > > > > 1, > > > > Skipped: > > > > 0, > > > > Time > > > > elapsed: > > > > 0 sec > > [junit] > > [junit] Testcase: > > > > > > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT:sanityCheck: > > > > Caused an ERROR > > [junit] Forked Java VM exited > > > > abnormally. > > > > Please > > > > note > > > > the > > > > time > > > > in > > > > the > > > > report does not reflect the time until the > > > > VM > > > > exit. > > > > [junit] > > > > junit.framework.AssertionFailedError: > > > > Forked > > > > Java > > > > VM > > > > exited > > > > abnormally. Please note the time in the > > > > report > > > > does > > > > not > > > > reflect > > > > the > > > > time > > > > until the VM exit. > > [junit] at > > > > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown > > > > Source) > > > > [junit] at > > > > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > [junit] at > > > > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown > > > > Source) > > > > [junit] at > > > > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > [junit] at > > > > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown > > > > Source) > > > > [junit] at > > > > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > [junit] at > > > > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown > > > > Source) > > > > [junit] at > > > > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > [junit] > > [junit] > > > > BUILD FAILED > > > > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: > > > > The > > > > following > > > > error occurred while executing this line: > > > > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: > > > > The > > > > following > > > > error occurred while executing this line: > > > > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102: > > > > Test > > > > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT > > > > failed > > > > (crashed) > > ``` > > > > > > 2023年10月13日(金) 21:56 Karl Wright< > > > > daddy...@gmail.com > > > > : > > > > r1912939 fixes this but I need to spin a > > > > new > > > > RC. > > > > Karl > > > > > > On Fri, Oct 13, 2023 at 8:46 AM Karl > > > > Wright < > > > > daddy...@gmail.com> > > > > wrote: > > > > Yes I get the same thing; a test needs > > > > to > > > > be > > > > updated. > > > > [javac] > > > > > > > C:\wip\mcf\release-2.26-branch\connectors\kafka\connector\src\test\java\org\apache\manifoldcf\agents\output\kafka\ZooKeeperLocal.java:45: > > > > error: unreported exception > > > > AdminServerException; > > > > must > > > > be > > > > caught or > > > > declared to be thrown > > [javac] > > > > zooKeeperServer.runFromConfig(configuration); > > > > [javac] > > > > Karl > > > > > > On Fri, Oct 13, 2023 at 8:35 AM Karl > > > > Wright > > > > < > > > > daddy...@gmail.com> > > > > wrote: > > > > There was a Zookeeper dependency change > > > > this > > > > release. I > > > > wonder if > > > > there > > > > is a test that needs to be updated. > > > > Let > > > > me > > > > try > > > > and > > > > see. > > > > Karl > > > > > > On Fri, Oct 13, 2023 at 4:51 AM > > > > Piergiorgio > > > > Lucidi > > > > < > > > > piergior...@apache.org> wrote: > > > > > > Hi Mingchun, > > > > thank you for your message and I was > > > > trying > > > > to > > > > build > > > > ManifoldCF > > > > using > > > > OpenJDK 17 so probably in the future > > > > for > > > > supporting > > > > this > > > > version > > > > of > > > > Java > > > > we > > should include Jaxb libraries as well. > > > > The build is ok now and I can compile > > > > and > > > > package > > > > everything > > > > correctly. > > > > Unfortunately executing tests I have > > > > the > > > > following > > > > error: > > > > compile-tests: > > [javac] Compiling 1 source file > > > > to > > > > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/build/connector-tests/classes > > > > [javac] > > > > > > > > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/connector/src/test/java/org/apache/manifoldcf/agents/output/kafka/ZooKeeperLocal.java:45: > > > > error: unreported exception > > > > AdminServerException; > > > > must > > > > be > > > > caught > > > > or > > > > declared to be thrown > > [javac] > > > > zooKeeperServer.runFromConfig(configuration); > > > > [javac] > > > > ^ > > > > [javac] 1 error > > > > BUILD FAILED > > > > > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:497: > > > > The following error occurred while > > > > executing > > > > this > > > > line: > > > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:471: > > > > The following error occurred while > > > > executing > > > > this > > > > line: > > > > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/dist/connector-build.xml:720: > > > > Compile failed; see the compiler error > > > > output > > > > for > > > > details. > > > > Any ideas? > > > > Thanks. > > > > Cheers, > > PG > > > > > > > > Il giorno gio 12 ott 2023 alle ore > > > > 10:39 > > > > Mingchun > > > > Zhao < > > > > mingchun.zha...@gmail.com> ha > > > > scritto: > > > > Hi Piergiorgio, > > > > FYI, Allow me to share the java and > > > > ant > > > > versions > > > > and > > > > the > > > > build > > > > steps > > > > I > > > > ran > > > > in my environment. > > > > ``` > > $ java --version > > openjdk 11.0.11 2021-04-20 > > OpenJDK Runtime Environment > > > > AdoptOpenJDK-11.0.11+9 > > > > (build > > > > 11.0.11+9) > > > > OpenJDK 64-Bit Server VM > > > > AdoptOpenJDK-11.0.11+9 > > > > (build > > > > 11.0.11+9, > > > > mixed > > > > mode) > > > > $ ant -version > > Apache Ant(TM) version 1.10.0 > > > > compiled > > > > on > > > > December > > > > 27 > > > > 2016 > > > > ``` > > ``` > > ant clean > > ant clean-deps > > ant clean-core-deps > > ant make-core-deps > > ant make-deps > > ant build > > ``` > > > > Regards, > > Mingchun > > > > 2023年10月12日(木) 17:32 Piergiorgio > > > > Lucidi < > > > > piergior...@apache.org > > > > : > > > > Hi folks, > > > > it seems that I can't compile the > > > > CswsConnector: > > > > [javac] public List<? extends > > > > AttributeGroup> > > > > getAttributeGroups() > > > > [javac] ^ > > [javac] symbol: class > > > > AttributeGroup > > > > [javac] location: class > > > > CswsConnector.ObjectInformation > > > > [javac] > > > > > > > > > /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:3966: > > > > error: cannot find symbol > > [javac] public > > > > NodePermissions > > > > getPermissions() > > > > [javac] ^ > > [javac] symbol: class > > > > NodePermissions > > > > [javac] location: class > > > > CswsConnector.ObjectInformation > > > > [javac] 100 errors > > [javac] 1 warning > > [javac] only showing the > > > > first > > > > 100 > > > > errors, > > > > of > > > > 123 > > > > total; > > > > use > > > > -Xmaxerrs > > > > if you would like to see more > > > > BUILD FAILED > > > > > > > /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:489: > > > > The following error occurred while > > > > executing > > > > this > > > > line: > > > > > > > /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:471: > > > > The following error occurred while > > > > executing > > > > this > > > > line: > > > > > /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/dist/connector-build.xml:686: > > > > Compile failed; see the compiler > > > > error > > > > output > > > > for > > > > details. > > > > Do you have any ideas? > > > > Thanks, > > PG > > > > Il giorno gio 12 ott 2023 alle ore > > > > 09:03 > > > > Guylaine > > > > BASSETTE < > > > > guylaine.basse...@francelabs.com> > > > > ha > > > > scritto: > > > > Hi, > > > > +1 from France Labs > > > > Regards, > > Guylaine > > > > France Labs – Your knowledge, now > > Datafari Enterprise Search – > > > > Découvrez > > > > la > > > > version > > > > 5 / > > > > Discover > > > > our > > > > version > > > > 5www.datafari.com < > > > > http://www.datafari.com> > > > > Le 11/10/2023 à 23:12, Furkan > > > > KAMACI > > > > a > > > > écrit > > > > : > > > > Hi, > > > > +1! > > > > Kind regards, > > Furkan Kamaci > > > > On Wed, Oct 11, 2023 at 11:13 PM > > > > Mingchun > > > > Zhao< > > > > mingchun.zha...@gmail.com > > > > wrote: > > > > > > Hi, > > Ran all the tests. > > +1 from me. > > > > Kind regards, > > Mingchun > > > > 2023年10月12日(木) 4:29 Cihad Guzel< > > > > cguz...@gmail.com > > > > : > > > > Hi, > > > > +1 from me > > > > Regards, > > Cihad Guzel > > > > 11 Eki 2023 Çar, saat 15:08 > > > > tarihinde > > > > Karl > > > > Wright< > > > > daddy...@gmail.com > > > > şunu > > yazdı: > > > > > > Please vote on whether to > > > > release > > > > Apache > > > > ManifoldCF > > > > 2.26, > > > > RC0. > > > > This release is the first > > > > release > > > > that > > > > requires > > > > at > > > > least > > > > Java > > > > 11, > > > > and > > > > it > > > > also includes a new CSV > > > > connector > > > > along > > > > with > > > > support > > > > for > > > > Solr > > > > 9. > > > > The > > > > release artifact can be found > > > > at: > > > > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.26 > > > > , > > > > and there is a release tag also > > > > at > > > > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC0 > > > > . > > > > Karl > > > > > > -- > > Piergiorgio > > > > > > -- > > Piergiorgio > > > > > > -- > > Cordialement, > > Guylaine > > > > France Labs – Your knowledge, now > > Datafari Enterprise Search – Découvrez la version > > > > 5 / > > > > Discover > > > > our > > > > version 5www.datafari.com <http://www.datafari.com> < > http://www.datafari.com> > > > > -- > > Cordialement, > > Guylaine > > > > France Labs – Your knowledge, now > > Datafari Enterprise Search – Découvrez la version 5 / > > > > Discover > > > > our > > > > version > > > > 5www.datafari.com <http://www.datafari.com> <http://www.datafari.com> > > > > -- > > Cordialement, > > Guylaine > > > > France Labs – Your knowledge, now > > Datafari Enterprise Search – Découvrez la version 5 / Discover our > > > > version > > > > 5www.datafari.com <http://www.datafari.com> <http://www.datafari.com> > > > > -- > > Cordialement, > > Guylaine > > > > France Labs – Your knowledge, now > > Datafari Enterprise Search – Découvrez la version 5 / Discover our > > version 5www.datafari.com <http://www.datafari.com> < > http://www.datafari.com> > > > > -- > > Cordialement, > > Guylaine > > > > France Labs – Your knowledge, now > > Datafari Enterprise Search – Découvrez la version 5 / Discover our > version > > 5 > > www.datafari.com > > >