Thanks. I've tried `ant test` with the latest trunk. As a result, the kafka
test failed as below.
```
[junit] Testcase:
sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
Caused an ERROR
[junit] io/netty/handler/ssl/SslContext
[junit] java.lang.NoClassDefFoundError: io/netty/handler/ssl/SslContext
[junit] at
org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96)
[junit] at
org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92)
[junit] at
org.apache.zookeeper.common.ZKConfig.<init>(ZKConfig.java:61)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.<init>(ZKClientConfig.java:69)
[junit] at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1620)
[junit] at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1603)
[junit] at
org.apache.manifoldcf.agents.output.kafka.KafkaLocal.<init>(KafkaLocal.java:31)
[junit] at
org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.setupKafka(BaseITHSQLDB.java:86)
[junit] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] Caused by: java.lang.ClassNotFoundException:
io.netty.handler.ssl.SslContext
[junit] at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
[junit] at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
[junit] at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
[junit]
[junit]
[junit] Testcase:
sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
Caused an ERROR
[junit] null
[junit] java.lang.NullPointerException
[junit] at
org.apache.manifoldcf.agents.output.kafka.BaseITHSQLDB.cleanUpKafka(BaseITHSQLDB.java:92)
[junit] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit]
[junit]
BUILD FAILED
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102:
Test org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT failed
```
2023年10月24日(火) 11:00 Karl Wright<daddy...@gmail.com>:
Okay, I updated zookeeper properly in build.xml and framework/build.xml,
with the two new dependencies, and the zookeeper tests pass. I haven't
tried the kafka or solr tests yet.
Karl
On Mon, Oct 23, 2023 at 9:29 PM Karl Wright<daddy...@gmail.com> wrote:
Unless I know what kafka is using zookeeper for, this would seem risky
to
me. Zookeeper is meant to coordinate processes; it may not work for
one
process to be using different versions of zookeeper than the others.
It looks like the original change to kafka you reverted had the proper
dependencies but they absolutely needed to be included in the right
classpaths and they weren't - they were only included in the kafka
tests.
I will look at this perhaps at the latest this weekend, but I won't
commit
this patch.
Karl
On Mon, Oct 23, 2023 at 5:14 PM Mingchun Zhao <
mingchun.zha...@gmail.com
wrote:
I reverted zookeeper version to 3.8.0 to avoid linkage error on the
multiThreadZooKeeperLockTest:
[junit] Caused by: java.lang.ClassNotFoundException:
io.netty.handler.ssl.SslContext
I've prepared a PR here:
https://github.com/apache/manifoldcf/pull/156
Just a heads up, `ant test` still hangs on the Solr Output connector
test:
```
run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
[junit] Configuration file successfully read
[junit] [main] INFO org.eclipse.jetty.util.log - Logging
initialized
@7027ms to org.eclipse.jetty.util.log.Slf4jLog
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0
Scavenging
every 660000ms
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler
-
Started o.e.j.w.WebAppContext@1517f633{ManifoldCF Crawler
Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-7554899724821045499/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-crawler-ui.war}
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler
-
Started o.e.j.w.WebAppContext@4fe01803{ManifoldCF Authorities API
Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-7701836901953162228/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-authority-service.war}
[junit] Creating mock service
[junit] Mock service created
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler
-
Started o.e.j.w.WebAppContext@13d186db{ManifoldCF General API
Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-2609388202403972652/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-api-service.war}
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@3bd55d8{HTTP/1.1, (http/1.1)}{0.0.0.0:8346}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started
@9054ms
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0
Scavenging
every 600000ms
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler
-
Started o.e.j.s.ServletContextHandler@6f4ade6e{/solr,null,AVAILABLE}
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@30e6a763{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started
@9064ms
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Stopped ServerConnector@30e6a763{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.session - node0
Stopped
scavenging
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler
-
Stopped o.e.j.s.ServletContextHandler@6f4ade6e{/solr,null,STOPPED}
```
2023年10月24日(火) 1:40 Karl Wright<daddy...@gmail.com>:
The dependencies would be in the zookeeper pom. Maven would follow
them
automatically which is why it is insufficient to assume that if
maven
works
so will ant.
You can use mvn dependency:tree to find what Maven is actually
pulling
in.
Karl
On Mon, Oct 23, 2023 at 11:04 AM Guylaine BASSETTE <
guylaine.basse...@francelabs.com> wrote:
I launch these tests with a Maven build and everything is OK. Yet
no
netty dependencies are requiered... I don't understand were this
SslContext is called...
Le 23/10/2023 à 16:25, Karl Wright a écrit :
Yes, that is indicating that zookeeper is looking for a specific
netty
class that it isn't finding. That is why I think there is now a
zookeeper
dependency we aren't including in the classpaths that include
zookeeper.
Karl
On Mon, Oct 23, 2023 at 10:23 AM Mingchun Zhao<
mingchun.zha...@gmail.com
wrote:
Karl, Thanks!
I think I reproduced that multiThreadZooKeeperLockTest error
when I
running `ant test`, will look into this.
```
[junit] ------------- ---------------- ---------------
[junit] Testcase:
multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks):
Caused an ERROR
[junit] io/netty/handler/ssl/SslContext
[junit] java.lang.NoClassDefFoundError:
io/netty/handler/ssl/SslContext
[junit] at
org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96)
[junit] at
org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92)
[junit] at
org.apache.zookeeper.common.ZKConfig.<init>(ZKConfig.java:61)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.<init>(ZKClientConfig.java:69)
[junit] at
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:643)
[junit] at
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:567)
[junit] at
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:734)
[junit] at
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:448)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.<init>(ZooKeeperConnection.java:66)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperConnectionPool.grab(ZooKeeperConnectionPool.java:48)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject.obtainGlobalReadLock(ZooKeeperLockObject.java:190)
[junit] at
org.apache.manifoldcf.core.lockmanager.LockObject.enterReadLock(LockObject.java:310)
[junit] at
org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:271)
[junit] at
org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks.enterReadLock(TestZooKeeperLocks.java:125)
[junit] at
org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks$ReaderThread.run(TestZooKeeperLocks.java:204)
[junit] Caused by: java.lang.ClassNotFoundException:
io.netty.handler.ssl.SslContext
[junit] at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
[junit] at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
[junit] at
java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
[junit]
[junit]
BUILD FAILED
```
2023年10月23日(月) 23:02 Karl Wright<daddy...@gmail.com>:
This is all you need to do:
ant clean-core-deps
ant make-core-deps
ant clean
ant test
Karl
On Mon, Oct 23, 2023 at 9:55 AM Mingchun Zhao <
mingchun.zha...@gmail.com
wrote:
Hi Guylaine, Thanks!
Thanks for all your sharing, it's very helpful! I'll
continue...
I'll look into it some more too. If I have any other
information
I'll
share
it with you.
2023年10月23日(月) 22:49 Guylaine BASSETTE <
guylaine.basse...@francelabs.com
:
Thanks for all your sharing, it's very helpful! I'll
continue...
Sorry, French and English mixed up!
Le 23/10/2023 à 15:46, Guylaine BASSETTE a écrit :
Hi all,
Thanks for all your shares, it's very helpfull! Merci pour
tous
vos
partages, c'est très utile ! Je poursuis...
Le 23/10/2023 à 15:31, Karl Wright a écrit :
I can't give advice on the test; this is something
FranceLabs
should
look
at.
However, nothing of what you are doing will affect the
zookeeper
tests in
framework. That's a totally different issue.
Karl
On Mon, Oct 23, 2023 at 9:20 AM Mingchun Zhao<
mingchun.zha...@gmail.com
wrote:
Hi Karl and Guylaine,
I hope and think it's just a problem specific to the
test.
Missing
updates or incompatible dependencies...
Allow me to share with you what I'm working on. I've
tried
to
support
http2C within the Solr output connector junit test, but
got
another
unhandled solr exception when I ran `ant run-IT-HSQLDB`.
- source code change
```
diff --git
a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests
/MockSolrService.java
b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output
/solr/tests/MockSolrService.java
index 237ade09c..3fb558f52 100644
---
a/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo
lrService.java
+++
b/connectors/solr/connector/src/test/java/org/apache/manifoldcf/agents/output/solr/tests/MockSo
lrService.java
@@ -18,7 +18,10 @@
*/
package
org.apache.manifoldcf.agents.output.solr.tests;
+import
org.eclipse.jetty.http2.server.HTTP2CServerConnectionFactory;
import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.server.HttpConfiguration;
+import org.eclipse.jetty.server.HttpConnectionFactory;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import
org.eclipse.jetty.servlet.ServletContextHandler;
@@ -40,7 +43,10 @@ public class MockSolrService
public MockSolrService()
{
server = new Server(new QueuedThreadPool(35));
- ServerConnector connector = new
ServerConnector(server);
+ HttpConfiguration config = new HttpConfiguration();
+ HttpConnectionFactory http1 = new
HttpConnectionFactory(config);
+ HTTP2CServerConnectionFactory http2c = new
HTTP2CServerConnectionFactory(config);
+ ServerConnector connector = new
ServerConnector(server,
http1,
http2c);
connector.setPort(8188);
server.addConnector(connector);
servlet = new SolrServlet();
@@ -111,6 +117,7 @@ public class MockSolrService
res.getWriter().printf(Locale.ROOT, "<solr>\n");
res.getWriter().printf(Locale.ROOT,
"</solr>\n");
res.getWriter().flush();
}
```
- run Junit test
```
~manifoldcf/connectors/solr% ant run-IT-HSQLDB
```
- confirm test-HSQLDB-output/manifoldcf.log
```
ERROR 2023-10-23T22:10:29,902 (Worker thread '24') -
Exception
tossed:
Unhandled Solr exception during indexinghttp://test70.txt
(200):
Error
from server athttp://localhost:8188/solr: Expected mime
type
application/octet-stream but got application/xml.
<result>
<doc name="something"/>
</result>
org.apache.manifoldcf.core.interfaces.ManifoldCFException:
Unhandled
Solr
exception during indexinghttp://test70.txt (200): Error
from
server
at
http://localhost:8188/solr: Expected mime type
application/octet-stream
but
got application/xml. <result>
<doc name="something"/>
</result>
at
org.apache.manifoldcf.agents.output.solr.HttpPoster.handleSolrException(HttpPoster.java:389)
~[classes/:?]
at
org.apache.manifoldcf.agents.output.solr.HttpPoster.indexPost(HttpPoster.java:544)
~[classes/:?]
at
org.apache.manifoldcf.agents.output.solr.SolrConnector.addOrReplaceDocumentWithException(SolrConnector.java:522)
~[classes/:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3214)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$OutputAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3395)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3065)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2696)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:750)
~[mcf-agents.jar:?]
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1585)
~[mcf-pull-agent.jar:?]
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1550)
~[mcf-pull-agent.jar:?]
at
org.apache.manifoldcf.crawler.tests.TestingRepositoryConnector.processDocuments(TestingRepositoryConnector.java:84)
~[mcf-pull-agent-tests.jar:?]
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:402)
~[mcf-pull-agent.jar:?]
ERROR 2023-10-23T22:10:29,902 (Worker thread '10') -
Exception
tossed:
Unhandled Solr exception during indexinghttp://test72.txt
(200):
Error
from server athttp://localhost:8188/solr: Expected mime
type
application/octet-stream but got application/xml.
<result>
<doc name="something"/>
</result>
```
Could you give me some advice?
2023年10月23日(月) 22:01 Mingchun Zhao<
mingchun.zha...@gmail.com
:
Then, wherever zookeeper is mentioned in
framework/build.xml,
a
reference to those dependencies must also be included.
It looks like zookeeper*.jar was already included in
connector-test-classpath within kafka/build.xml.
```
<path id="connector-test-classpath">
<path
refid="mcf-connector-build.connector-test-classpath"/>
<fileset dir="../../lib">
<include name="zookeeper*.jar"/>
<include name="kafka-clients*.jar"/>
<include name="lz4*.jar"/>
<include name="snappy-java*.jar"/>
<include name="metrics-core*.jar"/>
</fileset>
```
2023年10月23日(月) 21:50 Karl Wright<daddy...@gmail.com>:
Hi,
That just downloads zookeeper. But apparently the
zookeeper
version
required by Kafka now has dependencies of its own.
Otherwise
the
zookeeper
tests wouldn't fail with linkage errors.
The dependencies need to be identified and added in
several
places. The
first place is to the download-zookeeper part of the
root
build.xml
script. Then, wherever zookeeper is mentioned in
framework/build.xml, a
reference to those dependencies must also be included.
Karl
On Mon, Oct 23, 2023 at 8:32 AM Mingchun Zhao <
mingchun.zha...@gmail.com>
wrote:
Hi Karl,
Mingchun, did you add the jar(s) that the new
zookeeper
needs
to
the
build.xml download section?
Are the following settings correct? Or do you have an
old
version
of
zookeeper-*.jar left in your environment?
```build.xml
<target name="download-zookeeper">
<mkdir dir="lib"/>
<antcall target="download-via-maven">
<param name="target" value="lib"/>
<param name="project-path"
value="org/apache/zookeeper"/>
<param name="artifact-version"
value="${zookeeper.version}"/>
<param name="artifact-name"
value="zookeeper"/>
<param name="artifact-type"
value="jar"/>
</antcall>
... ...
```
Regards,
Mingchun
2023年10月23日(月) 21:19 Karl Wright<daddy...@gmail.com>:
Well, that was interesting.
Didn't get very far, because the dependency updates
committed
broke
Zookeeper again:
[junit] Testcase:
multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks):
Caused an ERROR
[junit] io/netty/handler/ssl/SslContext
[junit] java.lang.NoClassDefFoundError:
io/netty/handler/ssl/SslContext
[junit] at
org.apache.zookeeper.common.ZKConfig.handleBackwardCompatibility(ZKConfig.java:106)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.handleBackwardCompatibility(ZKClientConfig.java:96)
[junit] at
org.apache.zookeeper.common.ZKConfig.init(ZKConfig.java:92)
[junit] at
org.apache.zookeeper.common.ZKConfig.<init>(ZKConfig.java:61)
[junit] at
org.apache.zookeeper.client.ZKClientConfig.<init>(ZKClientConfig.java:69)
[junit] at
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:643)
[junit] at
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:567)
[junit] at
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:734)
[junit] at
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:448)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.createSession(ZooKeeperConnection.java:74)
[junit] at
org.apache.manifoldcf.core.lockmanager.ZooKeeperConnection.<init>(ZooKeeperConnection.java:66)
It looks like the even newer Zookeeper version has a
newer
dependency
that
isn't being included in the basic classpath, but
should
be.
Mingchun,
did
you add the jar(s) that the new zookeeper needs to
the
build.xml
download
section? If so, can you remind me what they were?
Karl
On Mon, Oct 23, 2023 at 8:11 AM Karl Wright<
daddy...@gmail.com
wrote:
I begin to suspect that the problem may be human
error.
If you don't do "ant clean-core-deps; ant
make-core-deps",
but
instead
just use "ant make-core-deps", you could have
incompatible
versions
of
several libraries in your classpath for the tests.
I'll
try
today
to
verify whether that might be happening by trying the
tests
locally
myself.
Karl
On Mon, Oct 23, 2023 at 7:57 AM Guylaine BASSETTE <
guylaine.basse...@francelabs.com> wrote:
Hi Karl and Mingchun,
Thanks for your work on the last few issues. I join
you on
this
Solr
testing problem.
That said, we've tested this new connector in our
application,
with a
FileShare job and everything was OK.
I hope and think it's just a problem specific to
the
test.
Missing
updates or incompatible dependencies...
Le 20/10/2023 à 02:58, Mingchun Zhao a écrit :
Hi Karl, Thanks!
so I wonder if, once again, there's a problem
with
dependencies
for
the
version of Solr they chose.
I'll take a look at this issue.
2023年10月20日(金) 9:50 Karl Wright<
daddy...@gmail.com
:
This connector FranceLabs updated. The problem
seems to
occur
at a
basic
level during http2 communication, so I wonder if,
once
again,
there's a
problem with dependencies for the version of Solr
they
chose.
Karl
On Thu, Oct 19, 2023 at 8:32 PM Mingchun Zhao<
mingchun.zha...@gmail.com>
wrote:
About the test "SolrCrawlHSQLDBIT" failure, it
seems
that
"IO
exception
during indexinghttp://test58.txt:
frame_size_error/invalid_frame_length"
error is occurring on the ManifoldCF side.
- command:
```
manifoldcf/connectors/solr% ant run-IT-HSQLDB
run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
... ...
```
- I checked
"connectors/solr/test-HSQLDB-output/manifoldcf.log":
```
WARN 2023-10-20T09:14:56,635 (Worker thread
'18') -
IO
exception
during
indexinghttp://test58.txt:
frame_size_error/invalid_frame_length
java.io.IOException:
frame_size_error/invalid_frame_length
at
org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:566)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session.access$2700(HTTP2Session.java:80)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:1857)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session$StreamsState.access$400(HTTP2Session.java:1436)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:511)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:506)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onConnectionFailure(Parser.java:414)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection$ParserListener.onConnectionFailure(HTTP2Connection.java:397)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:223)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:215)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.Parser.connectionFailure(Parser.java:209)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.Parser.parseHeader(Parser.java:155)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:121)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:261)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:183)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:138)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:361)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io
.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io
.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:69)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:350)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:271)
~[solr-solrj-9.1.0.jar:9.1.0
aa4f3d98ab19c201e7f3c74cd14c99174148616d
-
ishan - 2022-11-11 13:00:47]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
```
2023年10月19日(木) 20:37 Mingchun
Zhao<mingchun.zha...@gmail.com
:
I'm having trouble with the Solr test `ant
run-IT-HSQLDB`.
It's been over an hour since I ran the test and
it
still
doesn't
finish.
It seems that the problem is probably due to
the
Manifold
job
not
finishing.
Do you have any ideas or advice? Thanks!
```
~/ManifoldCF/manifoldcf/connectors/solr% ant
run-IT-HSQLDB
Buildfile:
/Users/zhaomingchun/ManifoldCF/manifoldcf/connectors/solr/build.xml
... ...
run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
[junit] Configuration file successfully
read
[junit] [main] INFO
org.eclipse.jetty.util.log
-
Logging
initialized
@7416ms to org.eclipse.jetty.util.log.Slf4jLog
[junit] [main] INFO
org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built:
2022-06-21T20:42:25.880Z;
git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm
11.0.11+9
[junit] [main] INFO
org.eclipse.jetty.server.session
-
DefaultSessionIdManager workerName=node0
[junit] [main] INFO
org.eclipse.jetty.server.session
-
No
SessionScavenger set, using defaults
[junit] [main] INFO
org.eclipse.jetty.server.session
-
node0
Scavenging every 600000ms
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@4bdc8b5d
{ManifoldCF
Crawler
Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-17189990303852051874/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-crawler-ui.war}
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@3bcd426c
{ManifoldCF
Authorities
API
Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-1541416273384984853/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-authority-service.war}
[junit] Creating mock service
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@5f14a673
{ManifoldCF
General
API
Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-12421572948842353797/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-api-service.war}
[junit] Mock service created
[junit] [main] INFO
org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@70325d20{HTTP/1.1,
(http/1.1)}{
0.0.0.0:8346}
[junit] [main] INFO
org.eclipse.jetty.server.Server -
Started
@9616ms
[junit] [main] INFO
org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built:
2022-06-21T20:42:25.880Z;
git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm
11.0.11+9
[junit] [main] INFO
org.eclipse.jetty.server.session
-
DefaultSessionIdManager workerName=node0
[junit] [main] INFO
org.eclipse.jetty.server.session
-
No
SessionScavenger set, using defaults
[junit] [main] INFO
org.eclipse.jetty.server.session
-
node0
Scavenging every 600000ms
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.s.ServletContextHandler@7808f638
{/solr,null,AVAILABLE}
[junit] [main] INFO
org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@2ab5afc7{HTTP/1.1,
(http/1.1)}{
0.0.0.0:8188}
[junit] [main] INFO
org.eclipse.jetty.server.Server -
Started
@9626ms
[junit] [main] INFO
org.eclipse.jetty.server.AbstractConnector -
Stopped ServerConnector@2ab5afc7{HTTP/1.1,
(http/1.1)}{
0.0.0.0:8188}
[junit] [main] INFO
org.eclipse.jetty.server.session
-
node0
Stopped
scavenging
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Stopped o.e.j.s.ServletContextHandler@7808f638
{/solr,null,STOPPED}
```
2023年10月19日(木) 20:05 Mingchun Zhao<
mingchun.zha...@gmail.com
:
Hi Karl,
I've tried to update Kafka and its
dependencies
to
the
latest
version
including zookeeper,
and confirmed tha kafka test run-IT-HSQLDB
passed
as
below:
```
~manifoldcf% cd connectors/kafka
~manifoldcf/connectors/kafka/% ant
run-IT-HSQLDB
BUILD SUCCESSFUL
Total time: 1 minute 19 seconds
```
Also, I prepared a PR for this issue:
https://github.com/apache/manifoldcf/pull/155
2023年10月19日(木) 7:12 Karl Wright<
daddy...@gmail.com>:
Hi,
It looks like the latest release of Kafka is
3.6.0.
I'd try setting that version in the pom for
connectors/kafka
and
doing
mvn
install. Then you can see what dependencies
it
wants
by:
mvn dependency:tree
It may be that Kafka no longer even requires
zookeeper
-
I
didn't
find
it
in a cursory inspection. But the
dependency:tree
would
be
the
final
word.
Karl
On Sat, Oct 14, 2023 at 2:17 AM Mingchun
Zhao <
mingchun.zha...@gmail.com
wrote:
Kalr, Thanks!
Though I'm not familiar with kafka, I'll try
to
find
out
what's
causing the
error as much as possible.
Kind Regards,
Mingchun
2023年10月14日(土) 14:07 Karl Wright<
daddy...@gmail.com
:
Yes, this seems to be something related to
zookeeper
update
and
the
Kafka
library version we're using.
Someone will need to dig into what is going
wrong
here
before
we
can
release. I don't know how widely used the
kafka
connector is
but
if
it
is
lightly used we can perhaps not distribute
the
connector
any
longer. But
that would be a last choice.
Karl
On Fri, Oct 13, 2023 at 12:12 PM Mingchun
Zhao
<
mingchun.zha...@gmail.com>
wrote:
By applying r1912939, I was able to
confirm
that
the
kafka
test
compile
error has disappeared when running `ant
test`.
Thanks, Karl!
However, the following error occurred on
subsequent
test
runs.
```
[junit] Testsuite:
org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT
[junit] Tests run: 1, Failures: 0, Errors:
1,
Skipped:
0,
Time
elapsed:
0 sec
[junit]
[junit] Testcase:
org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT:sanityCheck:
Caused an ERROR
[junit] Forked Java VM exited
abnormally.
Please
note
the
time
in
the
report does not reflect the time until the
VM
exit.
[junit]
junit.framework.AssertionFailedError:
Forked
Java
VM
exited
abnormally. Please note the time in the
report
does
not
reflect
the
time
until the VM exit.
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit]
[junit]
BUILD FAILED
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517:
The
following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471:
The
following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102:
Test
org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT
failed
(crashed)
```
2023年10月13日(金) 21:56 Karl Wright<
daddy...@gmail.com
:
r1912939 fixes this but I need to spin a
new
RC.
Karl
On Fri, Oct 13, 2023 at 8:46 AM Karl
Wright <
daddy...@gmail.com>
wrote:
Yes I get the same thing; a test needs
to
be
updated.
[javac]
C:\wip\mcf\release-2.26-branch\connectors\kafka\connector\src\test\java\org\apache\manifoldcf\agents\output\kafka\ZooKeeperLocal.java:45:
error: unreported exception
AdminServerException;
must
be
caught or
declared to be thrown
[javac]
zooKeeperServer.runFromConfig(configuration);
[javac]
Karl
On Fri, Oct 13, 2023 at 8:35 AM Karl
Wright
<
daddy...@gmail.com>
wrote:
There was a Zookeeper dependency change
this
release. I
wonder if
there
is a test that needs to be updated.
Let
me
try
and
see.
Karl
On Fri, Oct 13, 2023 at 4:51 AM
Piergiorgio
Lucidi
<
piergior...@apache.org> wrote:
Hi Mingchun,
thank you for your message and I was
trying
to
build
ManifoldCF
using
OpenJDK 17 so probably in the future
for
supporting
this
version
of
Java
we
should include Jaxb libraries as well.
The build is ok now and I can compile
and
package
everything
correctly.
Unfortunately executing tests I have
the
following
error:
compile-tests:
[javac] Compiling 1 source file
to
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/build/connector-tests/classes
[javac]
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/connector/src/test/java/org/apache/manifoldcf/agents/output/kafka/ZooKeeperLocal.java:45:
error: unreported exception
AdminServerException;
must
be
caught
or
declared to be thrown
[javac]
zooKeeperServer.runFromConfig(configuration);
[javac]
^
[javac] 1 error
BUILD FAILED
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:497:
The following error occurred while
executing
this
line:
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:471:
The following error occurred while
executing
this
line:
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/dist/connector-build.xml:720:
Compile failed; see the compiler error
output
for
details.
Any ideas?
Thanks.
Cheers,
PG
Il giorno gio 12 ott 2023 alle ore
10:39
Mingchun
Zhao <
mingchun.zha...@gmail.com> ha
scritto:
Hi Piergiorgio,
FYI, Allow me to share the java and
ant
versions
and
the
build
steps
I
ran
in my environment.
```
$ java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment
AdoptOpenJDK-11.0.11+9
(build
11.0.11+9)
OpenJDK 64-Bit Server VM
AdoptOpenJDK-11.0.11+9
(build
11.0.11+9,
mixed
mode)
$ ant -version
Apache Ant(TM) version 1.10.0
compiled
on
December
27
2016
```
```
ant clean
ant clean-deps
ant clean-core-deps
ant make-core-deps
ant make-deps
ant build
```
Regards,
Mingchun
2023年10月12日(木) 17:32 Piergiorgio
Lucidi <
piergior...@apache.org
:
Hi folks,
it seems that I can't compile the
CswsConnector:
[javac] public List<? extends
AttributeGroup>
getAttributeGroups()
[javac] ^
[javac] symbol: class
AttributeGroup
[javac] location: class
CswsConnector.ObjectInformation
[javac]
/Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:3966:
error: cannot find symbol
[javac] public
NodePermissions
getPermissions()
[javac] ^
[javac] symbol: class
NodePermissions
[javac] location: class
CswsConnector.ObjectInformation
[javac] 100 errors
[javac] 1 warning
[javac] only showing the
first
100
errors,
of
123
total;
use
-Xmaxerrs
if you would like to see more
BUILD FAILED
/Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:489:
The following error occurred while
executing
this
line:
/Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:471:
The following error occurred while
executing
this
line:
/Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/dist/connector-build.xml:686:
Compile failed; see the compiler
error
output
for
details.
Do you have any ideas?
Thanks,
PG
Il giorno gio 12 ott 2023 alle ore
09:03
Guylaine
BASSETTE <
guylaine.basse...@francelabs.com>
ha
scritto:
Hi,
+1 from France Labs
Regards,
Guylaine
France Labs – Your knowledge, now
Datafari Enterprise Search –
Découvrez
la
version
5 /
Discover
our
version
5
www.datafari.com <
http://www.datafari.com>
Le 11/10/2023 à 23:12, Furkan
KAMACI
a
écrit
:
Hi,
+1!
Kind regards,
Furkan Kamaci
On Wed, Oct 11, 2023 at 11:13 PM
Mingchun
Zhao<
mingchun.zha...@gmail.com
wrote:
Hi,
Ran all the tests.
+1 from me.
Kind regards,
Mingchun
2023年10月12日(木) 4:29 Cihad Guzel<
cguz...@gmail.com
:
Hi,
+1 from me
Regards,
Cihad Guzel
11 Eki 2023 Çar, saat 15:08
tarihinde
Karl
Wright<
daddy...@gmail.com
şunu
yazdı:
Please vote on whether to
release
Apache
ManifoldCF
2.26,
RC0.
This release is the first
release
that
requires
at
least
Java
11,
and
it
also includes a new CSV
connector
along
with
support
for
Solr
9.
The
release artifact can be found
at:
https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.26
,
and there is a release tag also
at
https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC0
.
Karl
--
Piergiorgio
--
Piergiorgio
--
Cordialement,
Guylaine
France Labs – Your knowledge, now
Datafari Enterprise Search – Découvrez la version
5 /
Discover
our
version 5
www.datafari.com <http://www.datafari.com>
--
Cordialement,
Guylaine
France Labs – Your knowledge, now
Datafari Enterprise Search – Découvrez la version 5 /
Discover
our
version
5
www.datafari.com <http://www.datafari.com>
--
Cordialement,
Guylaine
France Labs – Your knowledge, now
Datafari Enterprise Search – Découvrez la version 5 / Discover our
version
5
www.datafari.com <http://www.datafari.com>
