Hi Guylaine, Thanks! Was your patch based on the latest track? Unfortunately, I was unable to apply your patch in my environment.
2023年10月26日(木) 20:44 Guylaine BASSETTE <guylaine.basse...@francelabs.com>: > Hello Mingchun, > > As mentioned in my previous e-mail to Karl, my patch contained other files > to be updated. I don't know why they don't have been taken into account, > but would you be interested in trying my patch? I have fixed the spots I > had missed... (joint with this mail) > > With these modifications I went through all tests of kafka. > Le 26/10/2023 à 03:47, Mingchun Zhao a écrit : > > Hi there, Allow me to share my `ant test` result using the latest Github > trunk. > I got a lot of `Broker may not be available` warnings in the Kafka IT test > and it eventually failed. > > ``` > ... > [junit] [kafka-producer-network-thread | producer-3] INFO > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] > Node 0 disconnected. > [junit] [kafka-producer-network-thread | producer-3] WARN > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] > Connection to node 0 (/192.168.10.103:9092) could not be established. > Broker may not be available. > [junit] [kafka-producer-network-thread | producer-1] INFO > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] > Node 0 disconnected. > [junit] [kafka-producer-network-thread | producer-1] WARN > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] > Connection to node 0 (/192.168.10.103:9092) could not be established. > Broker may not be available. > [junit] [kafka-producer-network-thread | producer-2] INFO > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] > Node 0 disconnected. > [junit] [kafka-producer-network-thread | producer-2] WARN > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] > Connection to node 0 (/192.168.10.103:9092) could not be established. > Broker may not be available. > [junit] [kafka-producer-network-thread | producer-3] INFO > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] > Node 0 disconnected. > [junit] [kafka-producer-network-thread | producer-3] WARN > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] > Connection to node 0 (/192.168.10.103:9092) could not be established. > Broker may not be available. > [junit] ------------- ---------------- --------------- > [junit] Testcase: > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): > Caused an ERROR > [junit] ManifoldCF did not terminate in the allotted time of 120000 > milliseconds > [junit] org.apache.manifoldcf.core.interfaces.ManifoldCFException: > ManifoldCF did not terminate in the allotted time of 120000 milliseconds > [junit] at > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT.waitJobInactive(APISanityHSQLDBIT.java:289) > [junit] at > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT.sanityCheck(APISanityHSQLDBIT.java:177) > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > [junit] at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [junit] > [junit] > > BUILD FAILED > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following > error occurred while executing this line: > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following > error occurred while executing this line: > /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1104: > Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed > > Total time: 38 minutes 13 seconds > ``` > > > 2023年10月26日(木) 5:42 Karl Wright <daddy...@gmail.com> <daddy...@gmail.com>: > > > I see you have committed this. You missed a few spots; the extra jar was > mentioned in multiple places. I committed another fix to correct that. > > Karl > > > On Wed, Oct 25, 2023 at 10:46 AM Guylaine BASSETTE > <guylaine.basse...@francelabs.com> wrote: > > > Hello Karl, > > Thank you very much for this update! I have tested your suggestions and > kafka ITs tests ended successfully. :-) > > Here you can find the patch. > > My only doubt is this warning I had in some parts of the kafka test: > > ``` > > [junit] [Controller-0-to-broker-0-send-thread] INFO > org.apache.kafka.clients.NetworkClient - [Controller id=0, > targetBrokerId=0] Node 0 disconnected. > [junit] [Controller-0-to-broker-0-send-thread] WARN > org.apache.kafka.clients.NetworkClient - [Controller id=0, > targetBrokerId=0] Connection to node 0 > (guylaine-virtual-machine/127.0.1.1:9092) could not be established. Broker > may not be available. > [junit] [Controller-0-to-broker-0-send-thread] WARN > kafka.controller.RequestSendThread - [RequestSendThread controllerId=0] > Controller 0's connection to broker guylaine-virtual-machine:9092 (id: 0 > rack: null) was unsuccessful > [junit] java.io.IOException: Connection to > guylaine-virtual-machine:9092 (id: 0 rack: null) failed. > [junit] at > > > org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70) > > [junit] at > > > kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:298) > > [junit] at > > > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:251) > > [junit] at > > > org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130) > > [junit] [Controller-0-to-broker-0-send-thread] INFO > org.apache.kafka.clients.NetworkClient - [Controller id=0, > targetBrokerId=0] Client requested connection close from node 0 > [junit] [controller-event-thread] INFO state.change.logger - > [Controller id=0 epoch=2] Sending LeaderAndIsr request to broker 0 with 1 > become-leader and 0 become-follower partitions > ``` > > I have also ran an "ant test". It run core-framework and ITs tests until > mongoDB connector with this fail: > ``` > [junit] Testcase: > > > sanityCheck(org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT): > > Caused an ERROR > [junit] Could not start process: <EOF> > [junit] java.lang.RuntimeException: Could not start process: <EOF> > [junit] at > > > de.flapdoodle.embed.mongo.AbstractMongoProcess.onAfterProcessStart(AbstractMongoProcess.java:81) > > [junit] at > > > de.flapdoodle.embed.process.runtime.AbstractProcess.<init>(AbstractProcess.java:115) > > [junit] at > > > de.flapdoodle.embed.mongo.AbstractMongoProcess.<init>(AbstractMongoProcess.java:54) > > [junit] at > de.flapdoodle.embed.mongo.MongodProcess.<init>(MongodProcess.java:50) > [junit] at > > > de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:44) > > [junit] at > > > de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:34) > > [junit] at > de.flapdoodle.embed.process.runtime.Executable.start(Executable.java:109) > [junit] at > > > org.apache.manifoldcf.agents.output.mongodboutput.tests.BaseITHSQLDB.setUpMongoDB(BaseITHSQLDB.java:72) > > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit] at > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] > [junit] > [junit] Testcase: > > > sanityCheck(org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT): > > Caused an ERROR > [junit] null > [junit] java.lang.NullPointerException > [junit] at > > > org.apache.manifoldcf.agents.output.mongodboutput.tests.APISanityHSQLDBIT.removeTestArea(APISanityHSQLDBIT.java:109) > > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit] at > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > ``` > > And no more success for the ITs tests of Solr connector... > > Guylaine > Le 25/10/2023 à 12:52, Karl Wright a écrit : > > I was able to reproduce the problem last night. I believe the cause may > well be that we've moved too many dependencies to the framework level. > Specifically, I think perhaps only zookeeper and netty should run there, > but the Scala library probably needs to run at the same classloader level > as the other Scala jars, so it should be moved back into connectors/kafka > and removed from framework/build.xml and from build.xml and from > connector-build.xml. > > Sadly I'm completely snowed under until the weekend so it will need to > > wait > > until then unless someone else wants to try this. > > Karl > > > On Tue, Oct 24, 2023 at 9:12 AM Karl Wright <daddy...@gmail.com> > <daddy...@gmail.com> < > > daddy...@gmail.com> wrote: > > Try doing svn update and deleting your test-materials directory contents. > Then ant download-dependencies. You don't get a link error after that > > when > > you do run-IT-HSQLDB . > > Karl > > On Tue, Oct 24, 2023 at 9:10 AM Guylaine BASSETTE < > > guylaine.basse...@francelabs.com> wrote: > > Hi Karl and Mingchun, > > Thanks again ! > > My bad for the Zookeeper dependencies. Actually I've made a mistake > using in my IDE dependencies analyzer. > > Regarding Kafka tests, a light search brings me to the > "spark-streaming-kafka" dependency that might be missing... > > At the mean time, I continue my effort on Solr connector tests. > > > For any use, here is the errors I get: > > ``` > > [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler - > Stopped o.e.j.w.WebAppContext@7c28c1{ManifoldCF General API > > > > Webapp,/mcf-api-service,null,STOPPED}{/home/guylaine/IdeaProjects/mon-manifoldcf/dist/web/war/mcf-api-service.war} > > [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler > - Stopped o.e.j.w.WebAppContext@588ffeb{ManifoldCF Authorities API > > > > Webapp,/mcf-authority-service,null,STOPPED}{/home/guylaine/IdeaProjects/mon-manifoldcf/dist/web/war/mcf-authority-service.war} > > [junit] [main] INFO org.eclipse.jetty.server.handler.ContextHandler > - Stopped o.e.j.w.WebAppContext@71a3a190{ManifoldCF Crawler > > > > Interface,/mcf-crawler-ui,null,STOPPED}{/home/guylaine/IdeaProjects/mon-manifoldcf/dist/web/war/mcf-crawler-ui.war} > > [junit] ------------- ---------------- --------------- > [junit] Testcase: > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): > Caused an ERROR > [junit] 'scala.collection.immutable.ArraySeq > scala.runtime.ScalaRunTime$.wrapRefArray(java.lang.Object[])' > [junit] java.lang.NoSuchMethodError: > 'scala.collection.immutable.ArraySeq > scala.runtime.ScalaRunTime$.wrapRefArray(java.lang.Object[])' > [junit] at > kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala:338) > [junit] at > kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1603) > [junit] at > > > > org.apache.manifoldcf.agents.output.kafka.KafkaLocal.<init>(KafkaLocal.java:31) > > [junit] at > > > > org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.setupKafka(BaseITHSQLDB.java:86) > > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit] at > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] > [junit] > [junit] Testcase: > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): > Caused an ERROR > [junit] null > [junit] java.lang.NullPointerException > [junit] at > > > > org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.cleanUpKafka(BaseITHSQLDB.java:92) > > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit] at > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > ``` > > Le 24/10/2023 à 12:30, Karl Wright a écrit : > > I missed a place - connector-build.xml. Updated now. > > Now we don't get a link exception, but neither does the kafka test work. > It seems to be unable to start zookeeper even though all the > > dependencies > > are now there. Will need to look at this after work. > > Karl > > > On Mon, Oct 23, 2023 at 11:32 PM Mingchun Zhao< > mingchun.zha...@gmail.com> > > wrote: > > > Thanks. I've tried `ant test` with the latest trunk. As a result, the > > kafka > > test failed as below. > ``` > [junit] Testcase: > > > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): > > Caused an ERROR > [junit] io/netty/handler/ssl/SslContext > [junit] java.lang.NoClassDefFoundError: > > io/netty/handler/ssl/SslContext > > [junit] at > > > > > > org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106) > > [junit] at > > > > > > org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96) > > [junit] at > org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92) > [junit] at > org.apache.zookeeper.common.ZKConfig.<init>(ZKConfig.java:61) > [junit] at > > > org.apache.zookeeper.client.ZKClientConfig.<init>(ZKClientConfig.java:69) > > [junit] at > > kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1620) > > [junit] at > > kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1603) > > [junit] at > > > > > > org.apache.manifoldcf.agents.output.kafka.KafkaLocal.<init>(KafkaLocal.java:31) > > [junit] at > > > > > > org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.setupKafka(BaseITHSQLDB.java:86) > > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit] at > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] Caused by: java.lang.ClassNotFoundException: > io.netty.handler.ssl.SslContext > [junit] at > > > > > > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) > > [junit] at > > > > > > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) > > [junit] at > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) > [junit] > [junit] > [junit] Testcase: > > > sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT): > > Caused an ERROR > [junit] null > [junit] java.lang.NullPointerException > [junit] at > > > > > > org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.cleanUpKafka(BaseITHSQLDB.java:92) > > [junit] at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit] at > > > > > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [junit] at > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] > [junit] > > BUILD FAILED > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following > error occurred while executing this line: > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following > error occurred while executing this line: > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102: > > Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed > ``` > > 2023年10月24日(火) 11:00 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com> > <daddy...@gmail.com > : > > > Okay, I updated zookeeper properly in build.xml and > > framework/build.xml, > > with the two new dependencies, and the zookeeper tests pass. I > > haven't > > tried the kafka or solr tests yet. > > Karl > > > On Mon, Oct 23, 2023 at 9:29 PM Karl Wright<daddy...@gmail.com> > <daddy...@gmail.com> < > > daddy...@gmail.com> > > wrote: > > Unless I know what kafka is using zookeeper for, this would seem > > risky > > to > > me. Zookeeper is meant to coordinate processes; it may not work for > > one > > process to be using different versions of zookeeper than the others. > > It looks like the original change to kafka you reverted had the > > proper > > dependencies but they absolutely needed to be included in the right > classpaths and they weren't - they were only included in the kafka > > tests. > > I will look at this perhaps at the latest this weekend, but I won't > > commit > > this patch. > > > Karl > > > On Mon, Oct 23, 2023 at 5:14 PM Mingchun Zhao < > mingchun.zha...@gmail.com > > wrote: > > > I reverted zookeeper version to 3.8.0 to avoid linkage error on the > multiThreadZooKeeperLockTest: > [junit] Caused by: java.lang.ClassNotFoundException: > io.netty.handler.ssl.SslContext > > I've prepared a PR here:https://github.com/apache/manifoldcf/pull/156 > > Just a heads up, `ant test` still hangs on the Solr Output connector > > test: > > ``` > run-IT-HSQLDB: > [junit] Testsuite: > org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT > [junit] Configuration file successfully read > [junit] [main] INFO org.eclipse.jetty.util.log - Logging > > initialized > > @7027ms to org.eclipse.jetty.util.log.Slf4jLog > [junit] [main] INFO org.eclipse.jetty.server.Server - > jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: > 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9 > [junit] [main] INFO org.eclipse.jetty.server.session - > DefaultSessionIdManager workerName=node0 > [junit] [main] INFO org.eclipse.jetty.server.session - No > SessionScavenger set, using defaults > [junit] [main] INFO org.eclipse.jetty.server.session - node0 > Scavenging > every 660000ms > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler > > - > > Started o.e.j.w.WebAppContext@1517f633{ManifoldCF Crawler > > > > > > Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-7554899724821045499/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-crawler-ui.war} > > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler > > - > > Started o.e.j.w.WebAppContext@4fe01803{ManifoldCF Authorities API > > > > > > Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-7701836901953162228/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-authority-service.war} > > [junit] Creating mock service > [junit] Mock service created > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler > > - > > Started o.e.j.w.WebAppContext@13d186db{ManifoldCF General API > > > > > > Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-2609388202403972652/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-api-service.war} > > [junit] [main] INFO org.eclipse.jetty.server.AbstractConnector > > - > > Started ServerConnector@3bd55d8{HTTP/1.1, (http/1.1)}{0.0.0.0:8346} > [junit] [main] INFO org.eclipse.jetty.server.Server - Started > > @9054ms > > [junit] [main] INFO org.eclipse.jetty.server.Server - > jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: > 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9 > [junit] [main] INFO org.eclipse.jetty.server.session - > DefaultSessionIdManager workerName=node0 > [junit] [main] INFO org.eclipse.jetty.server.session - No > SessionScavenger set, using defaults > [junit] [main] INFO org.eclipse.jetty.server.session - node0 > Scavenging > every 600000ms > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler > > - > > Started o.e.j.s.ServletContextHandler@6f4ade6e > > {/solr,null,AVAILABLE} > > [junit] [main] INFO org.eclipse.jetty.server.AbstractConnector > > - > > Started ServerConnector@30e6a763{HTTP/1.1, (http/1.1)}{0.0.0.0:8188 > > } > > [junit] [main] INFO org.eclipse.jetty.server.Server - Started > > @9064ms > > [junit] [main] INFO org.eclipse.jetty.server.AbstractConnector > > - > > Stopped ServerConnector@30e6a763{HTTP/1.1, (http/1.1)}{0.0.0.0:8188 > > } > > [junit] [main] INFO org.eclipse.jetty.server.session - node0 > > Stopped > > scavenging > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler > > - > > Stopped o.e.j.s.ServletContextHandler@6f4ade6e{/solr,null,STOPPED} > ``` > > 2023年10月24日(火) 1:40 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com> > <daddy...@gmail.com > : > > > The dependencies would be in the zookeeper pom. Maven would follow > > them > > automatically which is why it is insufficient to assume that if > > maven > > works > > so will ant. > > You can use mvn dependency:tree to find what Maven is actually > > pulling > > in. > > Karl > > > On Mon, Oct 23, 2023 at 11:04 AM Guylaine BASSETTE < > > guylaine.basse...@francelabs.com> wrote: > > I launch these tests with a Maven build and everything is OK. Yet > > no > > netty dependencies are requiered... I don't understand were this > SslContext is called... > > Le 23/10/2023 à 16:25, Karl Wright a écrit : > > Yes, that is indicating that zookeeper is looking for a specific > > netty > > class that it isn't finding. That is why I think there is now a > > zookeeper > > dependency we aren't including in the classpaths that include > > zookeeper. > > Karl > > > On Mon, Oct 23, 2023 at 10:23 AM Mingchun Zhao< > mingchun.zha...@gmail.com > > wrote: > > > Karl, Thanks! > I think I reproduced that multiThreadZooKeeperLockTest error > > when I > > running `ant test`, will look into this. > ``` > [junit] ------------- ---------------- --------------- > [junit] Testcase: > > > > > > multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks): > > Caused an ERROR > [junit] io/netty/handler/ssl/SslContext > [junit] java.lang.NoClassDefFoundError: > > io/netty/handler/ssl/SslContext > > [junit] at > > > > > > org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106) > > [junit] at > > > > > > org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96) > > [junit] at > org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92) > [junit] at > org.apache.zookeeper.common.ZKConfig.<init>(ZKConfig.java:61) > [junit] at > > > org.apache.zookeeper.client.ZKClientConfig.<init>(ZKClientConfig.java:69) > > [junit] at > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:643) > [junit] at > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:567) > [junit] at > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:734) > [junit] at > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:448) > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74) > > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.<init>(ZooKeeperConnection.java:66) > > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnectionPool.grab(ZooKeeperConnectionPool.java:48) > > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.obtainGlobalReadLock(ZooKeeperLockObject.java:190) > > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.LockObject.enterReadLock(LockObject.java:310) > > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:271) > > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks.enterReadLock(TestZooKeeperLocks.java:125) > > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks$ReaderThread.run(TestZooKeeperLocks.java:204) > > [junit] Caused by: java.lang.ClassNotFoundException: > io.netty.handler.ssl.SslContext > [junit] at > > > > > > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) > > [junit] at > > > > > > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) > > [junit] at > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) > [junit] > [junit] > > BUILD FAILED > ``` > > 2023年10月23日(月) 23:02 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com> > <daddy...@gmail.com > : > > > This is all you need to do: > > ant clean-core-deps > ant make-core-deps > ant clean > ant test > > Karl > > > On Mon, Oct 23, 2023 at 9:55 AM Mingchun Zhao < > mingchun.zha...@gmail.com > > wrote: > > > Hi Guylaine, Thanks! > > > Thanks for all your sharing, it's very helpful! I'll > > continue... > > I'll look into it some more too. If I have any other > > information > > I'll > > share > > it with you. > > 2023年10月23日(月) 22:49 Guylaine BASSETTE < > guylaine.basse...@francelabs.com > > : > > > Thanks for all your sharing, it's very helpful! I'll > > continue... > > Sorry, French and English mixed up! > > Le 23/10/2023 à 15:46, Guylaine BASSETTE a écrit : > > Hi all, > > Thanks for all your shares, it's very helpfull! Merci pour > > tous > > vos > > partages, c'est très utile ! Je poursuis... > > Le 23/10/2023 à 15:31, Karl Wright a écrit : > > I can't give advice on the test; this is something > > FranceLabs > > should > > look > at. > However, nothing of what you are doing will affect the > > zookeeper > > tests in > framework. That's a totally different issue. > > Karl > > > On Mon, Oct 23, 2023 at 9:20 AM Mingchun Zhao< > mingchun.zha...@gmail.com > > wrote: > > > Hi Karl and Guylaine, > > > I hope and think it's just a problem specific to the > > test. > > Missing > > updates or incompatible dependencies... > > Allow me to share with you what I'm working on. I've > > tried > > to > > support > > http2C within the Solr output connector junit test, but > > got > > another > > unhandled solr exception when I ran `ant run-IT-HSQLDB`. > > - source code change > ``` > diff --git > > > > > > a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests > > /MockSolrService.java > > > > > > b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output > > /solr/tests/MockSolrService.java > index 237ade09c..3fb558f52 100644 > --- > > > > > > a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo > > lrService.java > +++ > > > > > > b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo > > lrService.java > @@ -18,7 +18,10 @@ > */ > package > > org.apache.manifoldcf.agents.output.solr.tests; > > +import > > org.eclipse.jetty.http2.server.HTTP2CServerConnectionFactory; > > import org.eclipse.jetty.servlet.ServletHolder; > +import org.eclipse.jetty.server.HttpConfiguration; > +import org.eclipse.jetty.server.HttpConnectionFactory; > import org.eclipse.jetty.server.Server; > import org.eclipse.jetty.server.ServerConnector; > import > > org.eclipse.jetty.servlet.ServletContextHandler; > > @@ -40,7 +43,10 @@ public class MockSolrService > public MockSolrService() > { > server = new Server(new QueuedThreadPool(35)); > - ServerConnector connector = new > > ServerConnector(server); > > + HttpConfiguration config = new HttpConfiguration(); > + HttpConnectionFactory http1 = new > > HttpConnectionFactory(config); > > + HTTP2CServerConnectionFactory http2c = new > HTTP2CServerConnectionFactory(config); > + ServerConnector connector = new > > ServerConnector(server, > > http1, > > http2c); > connector.setPort(8188); > server.addConnector(connector); > servlet = new SolrServlet(); > @@ -111,6 +117,7 @@ public class MockSolrService > res.getWriter().printf(Locale.ROOT, "<solr>\n"); > res.getWriter().printf(Locale.ROOT, > > "</solr>\n"); > > res.getWriter().flush(); > } > ``` > > - run Junit test > ``` > ~manifoldcf/connectors/solr% ant run-IT-HSQLDB > ``` > > - confirm test-HSQLDB-output/manifoldcf.log > ``` > ERROR 2023-10-23T22:10:29,902 (Worker thread '24') - > > Exception > > tossed: > > Unhandled Solr exception during indexinghttp://test70.txt > > (200): > > Error > > from server athttp://localhost:8188/solr: Expected mime > > type > > application/octet-stream but got application/xml. > > <result> > > <doc name="something"/> > </result> > > > > org.apache.manifoldcf.core.interfaces.ManifoldCFException: > > Unhandled > > Solr > exception during indexinghttp://test70.txt (200): Error > > from > > server > > at > http://localhost:8188/solr: Expected mime type > > application/octet-stream > > but > got application/xml. <result> > <doc name="something"/> > </result> > > at > > > > > > org.apache.manifoldcf.agents.output.solr.HttpPoster.handleSolrException(HttpPoster.java:389) > > ~[classes/:?] > at > > > > > > org.apache.manifoldcf.agents.output.solr.HttpPoster.indexPost(HttpPoster.java:544) > > ~[classes/:?] > at > > > > > > org.apache.manifoldcf.agents.output.solr.SolrConnector.addOrReplaceDocumentWithException(SolrConnector.java:522) > > ~[classes/:?] > at > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3214) > > ~[mcf-agents.jar:?] > at > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$OutputAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3395) > > ~[mcf-agents.jar:?] > at > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3065) > > ~[mcf-agents.jar:?] > at > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2696) > > ~[mcf-agents.jar:?] > at > > > > > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:750) > > ~[mcf-agents.jar:?] > at > > > > > > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1585) > > ~[mcf-pull-agent.jar:?] > at > > > > > > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1550) > > ~[mcf-pull-agent.jar:?] > at > > > > > > org.apache.manifoldcf.crawler.tests.TestingRepositoryConnector.processDocuments(TestingRepositoryConnector.java:84) > > ~[mcf-pull-agent-tests.jar:?] > at > > > > > > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:402) > > ~[mcf-pull-agent.jar:?] > ERROR 2023-10-23T22:10:29,902 (Worker thread '10') - > > Exception > > tossed: > > Unhandled Solr exception during indexinghttp://test72.txt > > (200): > > Error > > from server athttp://localhost:8188/solr: Expected mime > > type > > application/octet-stream but got application/xml. > > <result> > > <doc name="something"/> > </result> > ``` > > Could you give me some advice? > > 2023年10月23日(月) 22:01 Mingchun Zhao< > mingchun.zha...@gmail.com > > : > > Then, wherever zookeeper is mentioned in > > framework/build.xml, > > a > > reference to those dependencies must also be included. > > It looks like zookeeper*.jar was already included in > connector-test-classpath within kafka/build.xml. > ``` > <path id="connector-test-classpath"> > <path > > refid="mcf-connector-build.connector-test-classpath"/> > > <fileset dir="../../lib"> > <include name="zookeeper*.jar"/> > <include name="kafka-clients*.jar"/> > <include name="lz4*.jar"/> > <include name="snappy-java*.jar"/> > <include name="metrics-core*.jar"/> > </fileset> > ``` > > > 2023年10月23日(月) 21:50 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com> > <daddy...@gmail.com > : > > > Hi, > > That just downloads zookeeper. But apparently the > > zookeeper > > version > > required by Kafka now has dependencies of its own. > > Otherwise > > the > > zookeeper > tests wouldn't fail with linkage errors. > > The dependencies need to be identified and added in > > several > > places. The > first place is to the download-zookeeper part of the > > root > > build.xml > > script. Then, wherever zookeeper is mentioned in > framework/build.xml, a > reference to those dependencies must also be included. > > Karl > > > On Mon, Oct 23, 2023 at 8:32 AM Mingchun Zhao < > mingchun.zha...@gmail.com> > > wrote: > > > Hi Karl, > > > Mingchun, did you add the jar(s) that the new > > zookeeper > > needs > > to > > the > > build.xml download section? > > Are the following settings correct? Or do you have an > > old > > version > > of > > zookeeper-*.jar left in your environment? > > ```build.xml > <target name="download-zookeeper"> > <mkdir dir="lib"/> > <antcall target="download-via-maven"> > <param name="target" value="lib"/> > <param name="project-path" > value="org/apache/zookeeper"/> > <param name="artifact-version" > > value="${zookeeper.version}"/> > > <param name="artifact-name" > > value="zookeeper"/> > > <param name="artifact-type" > > value="jar"/> > > </antcall> > ... ... > ``` > > Regards, > Mingchun > > > 2023年10月23日(月) 21:19 Karl Wright<daddy...@gmail.com> <daddy...@gmail.com> > <daddy...@gmail.com > : > > > Well, that was interesting. > > Didn't get very far, because the dependency updates > > committed > > broke > > Zookeeper again: > > [junit] Testcase: > > > > > > multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks): > > Caused an ERROR > [junit] io/netty/handler/ssl/SslContext > [junit] java.lang.NoClassDefFoundError: > > io/netty/handler/ssl/SslContext > > [junit] at > > > > > > org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106) > > [junit] at > > > > > > org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96) > > [junit] at > > > org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92) > > [junit] at > > > org.apache.zookeeper.common.ZKConfig.<init>(ZKConfig.java:61) > > [junit] at > > > org.apache.zookeeper.client.ZKClientConfig.<init>(ZKClientConfig.java:69) > > [junit] at > > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:643) > > [junit] at > > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:567) > > [junit] at > > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:734) > > [junit] at > > > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:448) > > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74) > > [junit] at > > > > > > org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.<init>(ZooKeeperConnection.java:66) > > It looks like the even newer Zookeeper version has a > > newer > > dependency > > that > > isn't being included in the basic classpath, but > > should > > be. > > Mingchun, > > did > > you add the jar(s) that the new zookeeper needs to > > the > > build.xml > > download > > section? If so, can you remind me what they were? > > Karl > > > On Mon, Oct 23, 2023 at 8:11 AM Karl Wright< > daddy...@gmail.com > > wrote: > > I begin to suspect that the problem may be human > > error. > > If you don't do "ant clean-core-deps; ant > > make-core-deps", > > but > > instead > > just use "ant make-core-deps", you could have > > incompatible > > versions > > of > > several libraries in your classpath for the tests. > > I'll > > try > > today > > to > > verify whether that might be happening by trying the > > tests > > locally > > myself. > > Karl > > > On Mon, Oct 23, 2023 at 7:57 AM Guylaine BASSETTE < > > guylaine.basse...@francelabs.com> wrote: > > Hi Karl and Mingchun, > > Thanks for your work on the last few issues. I join > > you on > > this > > Solr > > testing problem. > > That said, we've tested this new connector in our > > application, > > with a > > FileShare job and everything was OK. > > I hope and think it's just a problem specific to > > the > > test. > > Missing > > updates or incompatible dependencies... > > Le 20/10/2023 à 02:58, Mingchun Zhao a écrit : > > Hi Karl, Thanks! > > > so I wonder if, once again, there's a problem > > with > > dependencies > > for > > the > > version of Solr they chose. > > I'll take a look at this issue. > > 2023年10月20日(金) 9:50 Karl Wright< > daddy...@gmail.com > > : > > This connector FranceLabs updated. The problem > > seems to > > occur > > at a > > basic > > level during http2 communication, so I wonder if, > > once > > again, > > there's a > > problem with dependencies for the version of Solr > > they > > chose. > > Karl > > > On Thu, Oct 19, 2023 at 8:32 PM Mingchun Zhao< > mingchun.zha...@gmail.com> > > wrote: > > > About the test "SolrCrawlHSQLDBIT" failure, it > > seems > > that > > "IO > > exception > > during indexinghttp://test58.txt: > > frame_size_error/invalid_frame_length" > > error is occurring on the ManifoldCF side. > > - command: > ``` > manifoldcf/connectors/solr% ant run-IT-HSQLDB > > run-IT-HSQLDB: > [junit] Testsuite: > > > org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT > > ... ... > ``` > > - I checked > > "connectors/solr/test-HSQLDB-output/manifoldcf.log": > > ``` > WARN 2023-10-20T09:14:56,635 (Worker thread > > '18') - > > IO > > exception > > during > > indexinghttp://test58.txt: > > frame_size_error/invalid_frame_length > > java.io.IOException: > > frame_size_error/invalid_frame_length > > at > > org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:566) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.http2.HTTP2Session.access$2700(HTTP2Session.java:80) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:1857) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.HTTP2Session$StreamsState.access$400(HTTP2Session.java:1436) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:511) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:506) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onConnectionFailure(Parser.java:414) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.HTTP2Connection$ParserListener.onConnectionFailure(HTTP2Connection.java:397) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:223) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:215) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.http2.parser.Parser.connectionFailure(Parser.java:209) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.http2.parser.Parser.parseHeader(Parser.java:155) > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:121) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:261) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362) > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186) > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:183) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:138) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:361) > > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at org.eclipse.jetty.io > > .FillInterest.fillable(FillInterest.java:105) > > ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622] > > at org.eclipse.jetty.io > > .ChannelEndPoint$1.run(ChannelEndPoint.java:104) > > ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:69) > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:350) > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305) > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > > > > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:271) > > ~[solr-solrj-9.1.0.jar:9.1.0 > > aa4f3d98ab19c201e7f3c74cd14c99174148616d > > - > > ishan - 2022-11-11 13:00:47] > at > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > ~[?:?] > at > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > ~[?:?] > at java.lang.Thread.run(Thread.java:829) ~[?:?] > ``` > > 2023年10月19日(木) 20:37 Mingchun > Zhao<mingchun.zha...@gmail.com > > : > > I'm having trouble with the Solr test `ant > > run-IT-HSQLDB`. > > It's been over an hour since I ran the test and > > it > > still > > doesn't > > finish. > > It seems that the problem is probably due to > > the > > Manifold > > job > > not > > finishing. > Do you have any ideas or advice? Thanks! > > ``` > ~/ManifoldCF/manifoldcf/connectors/solr% ant > > run-IT-HSQLDB > > Buildfile: > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/connectors/solr/build.xml > > ... ... > run-IT-HSQLDB: > [junit] Testsuite: > > > org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT > > [junit] Configuration file successfully > > read > > [junit] [main] INFO > > org.eclipse.jetty.util.log > > - > > Logging > > initialized > > @7416ms to org.eclipse.jetty.util.log.Slf4jLog > [junit] [main] INFO > > org.eclipse.jetty.server.Server - > > jetty-9.4.48.v20220622; built: > > 2022-06-21T20:42:25.880Z; > > git: > > 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm > > 11.0.11+9 > > [junit] [main] INFO > > org.eclipse.jetty.server.session > > - > > DefaultSessionIdManager workerName=node0 > [junit] [main] INFO > > org.eclipse.jetty.server.session > > - > > No > > SessionScavenger set, using defaults > [junit] [main] INFO > > org.eclipse.jetty.server.session > > - > > node0 > > Scavenging every 600000ms > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler - > > Started o.e.j.w.WebAppContext@4bdc8b5d > > {ManifoldCF > > Crawler > > > > Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-17189990303852051874/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-crawler-ui.war} > > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler - > > Started o.e.j.w.WebAppContext@3bcd426c > > {ManifoldCF > > Authorities > > API > > > > Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-1541416273384984853/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-authority-service.war} > > [junit] Creating mock service > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler - > > Started o.e.j.w.WebAppContext@5f14a673 > > {ManifoldCF > > General > > API > > > > > Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-12421572948842353797/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-api-service.war} > > [junit] Mock service created > [junit] [main] INFO > > org.eclipse.jetty.server.AbstractConnector - > > Started ServerConnector@70325d20{HTTP/1.1, > > (http/1.1)}{ > 0.0.0.0:8346} > > [junit] [main] INFO > > org.eclipse.jetty.server.Server - > > Started > > @9616ms > > [junit] [main] INFO > > org.eclipse.jetty.server.Server - > > jetty-9.4.48.v20220622; built: > > 2022-06-21T20:42:25.880Z; > > git: > > 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm > > 11.0.11+9 > > [junit] [main] INFO > > org.eclipse.jetty.server.session > > - > > DefaultSessionIdManager workerName=node0 > [junit] [main] INFO > > org.eclipse.jetty.server.session > > - > > No > > SessionScavenger set, using defaults > [junit] [main] INFO > > org.eclipse.jetty.server.session > > - > > node0 > > Scavenging every 600000ms > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler - > > Started o.e.j.s.ServletContextHandler@7808f638 > > {/solr,null,AVAILABLE} > > [junit] [main] INFO > > org.eclipse.jetty.server.AbstractConnector - > > Started ServerConnector@2ab5afc7{HTTP/1.1, > > (http/1.1)}{ > 0.0.0.0:8188} > > [junit] [main] INFO > > org.eclipse.jetty.server.Server - > > Started > > @9626ms > > [junit] [main] INFO > > org.eclipse.jetty.server.AbstractConnector - > > Stopped ServerConnector@2ab5afc7{HTTP/1.1, > > (http/1.1)}{ > 0.0.0.0:8188} > > [junit] [main] INFO > > org.eclipse.jetty.server.session > > - > > node0 > > Stopped > > scavenging > [junit] [main] INFO > > org.eclipse.jetty.server.handler.ContextHandler - > > Stopped o.e.j.s.ServletContextHandler@7808f638 > > {/solr,null,STOPPED} > > ``` > > 2023年10月19日(木) 20:05 Mingchun Zhao< > mingchun.zha...@gmail.com > > : > > Hi Karl, > > I've tried to update Kafka and its > > dependencies > > to > > the > > latest > > version > > including zookeeper, > and confirmed tha kafka test run-IT-HSQLDB > > passed > > as > > below: > > ``` > ~manifoldcf% cd connectors/kafka > ~manifoldcf/connectors/kafka/% ant > > run-IT-HSQLDB > > BUILD SUCCESSFUL > Total time: 1 minute 19 seconds > ``` > > Also, I prepared a PR for this issue: > > https://github.com/apache/manifoldcf/pull/155 > > 2023年10月19日(木) 7:12 Karl Wright< > daddy...@gmail.com>: > > Hi, > > It looks like the latest release of Kafka is > > 3.6.0. > > I'd try setting that version in the pom for > > connectors/kafka > > and > > doing > > mvn > install. Then you can see what dependencies > > it > > wants > > by: > > mvn dependency:tree > > It may be that Kafka no longer even requires > > zookeeper > > - > > I > > didn't > > find > > it > > in a cursory inspection. But the > > dependency:tree > > would > > be > > the > > final > > word. > > Karl > > > On Sat, Oct 14, 2023 at 2:17 AM Mingchun > > Zhao < > mingchun.zha...@gmail.com > > wrote: > > > Kalr, Thanks! > Though I'm not familiar with kafka, I'll try > > to > > find > > out > > what's > > causing the > > error as much as possible. > > Kind Regards, > Mingchun > > > 2023年10月14日(土) 14:07 Karl Wright< > daddy...@gmail.com > > : > > Yes, this seems to be something related to > > zookeeper > > update > > and > > the > > Kafka > > library version we're using. > > Someone will need to dig into what is going > > wrong > > here > > before > > we > > can > > release. I don't know how widely used the > > kafka > > connector is > > but > > if > > it > > is > > lightly used we can perhaps not distribute > > the > > connector > > any > > longer. But > > that would be a last choice. > > Karl > > > On Fri, Oct 13, 2023 at 12:12 PM Mingchun > > Zhao > > < > mingchun.zha...@gmail.com> > > wrote: > > > By applying r1912939, I was able to > > confirm > > that > > the > > kafka > > test > > compile > > error has disappeared when running `ant > > test`. > > Thanks, Karl! > > However, the following error occurred on > > subsequent > > test > > runs. > > ``` > [junit] Testsuite: > > > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT > > [junit] Tests run: 1, Failures: 0, Errors: > > 1, > > Skipped: > > 0, > > Time > > elapsed: > > 0 sec > [junit] > [junit] Testcase: > > > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT:sanityCheck: > > Caused an ERROR > [junit] Forked Java VM exited > > abnormally. > > Please > > note > > the > > time > > in > > the > > report does not reflect the time until the > > VM > > exit. > > [junit] > > junit.framework.AssertionFailedError: > > Forked > > Java > > VM > > exited > > abnormally. Please note the time in the > > report > > does > > not > > reflect > > the > > time > > until the VM exit. > [junit] at > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown > > Source) > > [junit] at > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] at > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown > > Source) > > [junit] at > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] at > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown > > Source) > > [junit] at > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] at > > > jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown > > Source) > > [junit] at > > > > > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [junit] > [junit] > > BUILD FAILED > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: > > The > > following > > error occurred while executing this line: > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: > > The > > following > > error occurred while executing this line: > > > /Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102: > > Test > > org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT > > failed > > (crashed) > ``` > > > 2023年10月13日(金) 21:56 Karl Wright< > daddy...@gmail.com > > : > > r1912939 fixes this but I need to spin a > > new > > RC. > > Karl > > > On Fri, Oct 13, 2023 at 8:46 AM Karl > > Wright < > daddy...@gmail.com> > > wrote: > > Yes I get the same thing; a test needs > > to > > be > > updated. > > [javac] > > > > > C:\wip\mcf\release-2.26-branch\connectors\kafka\connector\src\test\java\org\apache\manifoldcf\agents\output\kafka\ZooKeeperLocal.java:45: > > error: unreported exception > > AdminServerException; > > must > > be > > caught or > > declared to be thrown > [javac] > > zooKeeperServer.runFromConfig(configuration); > > [javac] > > Karl > > > On Fri, Oct 13, 2023 at 8:35 AM Karl > > Wright > > < > daddy...@gmail.com> > > wrote: > > There was a Zookeeper dependency change > > this > > release. I > > wonder if > > there > > is a test that needs to be updated. > > Let > > me > > try > > and > > see. > > Karl > > > On Fri, Oct 13, 2023 at 4:51 AM > > Piergiorgio > > Lucidi > > < > piergior...@apache.org> wrote: > > > Hi Mingchun, > > thank you for your message and I was > > trying > > to > > build > > ManifoldCF > > using > > OpenJDK 17 so probably in the future > > for > > supporting > > this > > version > > of > > Java > > we > should include Jaxb libraries as well. > > The build is ok now and I can compile > > and > > package > > everything > > correctly. > > Unfortunately executing tests I have > > the > > following > > error: > > compile-tests: > [javac] Compiling 1 source file > > to > > > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/build/connector-tests/classes > > [javac] > > > > > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/connector/src/test/java/org/apache/manifoldcf/agents/output/kafka/ZooKeeperLocal.java:45: > > error: unreported exception > > AdminServerException; > > must > > be > > caught > > or > > declared to be thrown > [javac] > > zooKeeperServer.runFromConfig(configuration); > > [javac] > > ^ > > [javac] 1 error > > BUILD FAILED > > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:497: > > The following error occurred while > > executing > > this > > line: > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:471: > > The following error occurred while > > executing > > this > > line: > > > > /Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/dist/connector-build.xml:720: > > Compile failed; see the compiler error > > output > > for > > details. > > Any ideas? > > Thanks. > > Cheers, > PG > > > > Il giorno gio 12 ott 2023 alle ore > > 10:39 > > Mingchun > > Zhao < > mingchun.zha...@gmail.com> ha > > scritto: > > Hi Piergiorgio, > > FYI, Allow me to share the java and > > ant > > versions > > and > > the > > build > > steps > > I > > ran > > in my environment. > > ``` > $ java --version > openjdk 11.0.11 2021-04-20 > OpenJDK Runtime Environment > > AdoptOpenJDK-11.0.11+9 > > (build > > 11.0.11+9) > > OpenJDK 64-Bit Server VM > > AdoptOpenJDK-11.0.11+9 > > (build > > 11.0.11+9, > > mixed > > mode) > > $ ant -version > Apache Ant(TM) version 1.10.0 > > compiled > > on > > December > > 27 > > 2016 > > ``` > ``` > ant clean > ant clean-deps > ant clean-core-deps > ant make-core-deps > ant make-deps > ant build > ``` > > Regards, > Mingchun > > 2023年10月12日(木) 17:32 Piergiorgio > > Lucidi < > piergior...@apache.org > > : > > Hi folks, > > it seems that I can't compile the > > CswsConnector: > > [javac] public List<? extends > > AttributeGroup> > > getAttributeGroups() > > [javac] ^ > [javac] symbol: class > > AttributeGroup > > [javac] location: class > > CswsConnector.ObjectInformation > > [javac] > > > > > > /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:3966: > > error: cannot find symbol > [javac] public > > NodePermissions > > getPermissions() > > [javac] ^ > [javac] symbol: class > > NodePermissions > > [javac] location: class > > CswsConnector.ObjectInformation > > [javac] 100 errors > [javac] 1 warning > [javac] only showing the > > first > > 100 > > errors, > > of > > 123 > > total; > > use > > -Xmaxerrs > > if you would like to see more > > BUILD FAILED > > > > > /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:489: > > The following error occurred while > > executing > > this > > line: > > > > > /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:471: > > The following error occurred while > > executing > > this > > line: > > > > /Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/dist/connector-build.xml:686: > > Compile failed; see the compiler > > error > > output > > for > > details. > > Do you have any ideas? > > Thanks, > PG > > Il giorno gio 12 ott 2023 alle ore > > 09:03 > > Guylaine > > BASSETTE < > guylaine.basse...@francelabs.com> > > ha > > scritto: > > Hi, > > +1 from France Labs > > Regards, > Guylaine > > France Labs – Your knowledge, now > Datafari Enterprise Search – > > Découvrez > > la > > version > > 5 / > > Discover > > our > > version > 5www.datafari.com < > > http://www.datafari.com> <http://www.datafari.com> > > Le 11/10/2023 à 23:12, Furkan > > KAMACI > > a > > écrit > > : > > Hi, > > +1! > > Kind regards, > Furkan Kamaci > > On Wed, Oct 11, 2023 at 11:13 PM > > Mingchun > > Zhao< > mingchun.zha...@gmail.com > > wrote: > > > Hi, > Ran all the tests. > +1 from me. > > Kind regards, > Mingchun > > 2023年10月12日(木) 4:29 Cihad Guzel< > cguz...@gmail.com > > : > > Hi, > > +1 from me > > Regards, > Cihad Guzel > > 11 Eki 2023 Çar, saat 15:08 > > tarihinde > > Karl > > Wright< > daddy...@gmail.com > > şunu > yazdı: > > > Please vote on whether to > > release > > Apache > > ManifoldCF > > 2.26, > > RC0. > > This release is the first > > release > > that > > requires > > at > > least > > Java > > 11, > > and > > it > > also includes a new CSV > > connector > > along > > with > > support > > for > > Solr > > 9. > > The > > release artifact can be found > > at: > https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.26 > > , > > and there is a release tag also > > at > https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC0 > > . > > Karl > > > -- > Piergiorgio > > > -- > Piergiorgio > > > -- > Cordialement, > Guylaine > > France Labs – Your knowledge, now > Datafari Enterprise Search – Découvrez la version > > 5 / > > Discover > > our > > version 5www.datafari.com <http://www.datafari.com> > <http://www.datafari.com> < > > http://www.datafari.com> > > -- > Cordialement, > Guylaine > > France Labs – Your knowledge, now > Datafari Enterprise Search – Découvrez la version 5 / > > Discover > > our > > version > 5www.datafari.com <http://www.datafari.com> <http://www.datafari.com> > <http://www.datafari.com> <http://www.datafari.com> > > -- > Cordialement, > Guylaine > > France Labs – Your knowledge, now > Datafari Enterprise Search – Découvrez la version 5 / Discover our > > version > 5www.datafari.com <http://www.datafari.com> <http://www.datafari.com> > <http://www.datafari.com> <http://www.datafari.com> > > -- > Cordialement, > Guylaine > > France Labs – Your knowledge, now > Datafari Enterprise Search – Découvrez la version 5 / Discover our > version 5www.datafari.com <http://www.datafari.com> <http://www.datafari.com> > < > > http://www.datafari.com> > > -- > Cordialement, > Guylaine > > France Labs – Your knowledge, now > Datafari Enterprise Search – Découvrez la version 5 / Discover our > > version > > 5www.datafari.com > > -- > Cordialement, > Guylaine > > France Labs – Your knowledge, now > Datafari Enterprise Search – Découvrez la version 5 / Discover our version > 5 > www.datafari.com >