This connector FranceLabs updated. The problem seems to occur at a basic
level during http2 communication, so I wonder if, once again, there's a
problem with dependencies for the version of Solr they chose.
Karl
On Thu, Oct 19, 2023 at 8:32 PM Mingchun Zhao<mingchun.zha...@gmail.com>
wrote:
About the test "SolrCrawlHSQLDBIT" failure, it seems that "IO exception
during indexinghttp://test58.txt:
frame_size_error/invalid_frame_length"
error is occurring on the ManifoldCF side.
- command:
```
manifoldcf/connectors/solr% ant run-IT-HSQLDB
run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
... ...
```
- I checked "connectors/solr/test-HSQLDB-output/manifoldcf.log":
```
WARN 2023-10-20T09:14:56,635 (Worker thread '18') - IO exception during
indexinghttp://test58.txt: frame_size_error/invalid_frame_length
java.io.IOException: frame_size_error/invalid_frame_length
at org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:566)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.http2.HTTP2Session.access$2700(HTTP2Session.java:80)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:1857)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session$StreamsState.access$400(HTTP2Session.java:1436)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:511)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:506)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onConnectionFailure(Parser.java:414)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection$ParserListener.onConnectionFailure(HTTP2Connection.java:397)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:223)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:215)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.parser.Parser.connectionFailure(Parser.java:209)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.http2.parser.Parser.parseHeader(Parser.java:155)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:121)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:261)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:183)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:138)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:361)
~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:69)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:350)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:271)
~[solr-solrj-9.1.0.jar:9.1.0 aa4f3d98ab19c201e7f3c74cd14c99174148616d -
ishan - 2022-11-11 13:00:47]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
```
2023年10月19日(木) 20:37 Mingchun Zhao<mingchun.zha...@gmail.com>:
I'm having trouble with the Solr test `ant run-IT-HSQLDB`.
It's been over an hour since I ran the test and it still doesn't
finish.
It seems that the problem is probably due to the Manifold job not
finishing.
Do you have any ideas or advice? Thanks!
```
~/ManifoldCF/manifoldcf/connectors/solr% ant run-IT-HSQLDB
Buildfile:
/Users/zhaomingchun/ManifoldCF/manifoldcf/connectors/solr/build.xml
... ...
run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
[junit] Configuration file successfully read
[junit] [main] INFO org.eclipse.jetty.util.log - Logging
initialized
@7416ms to org.eclipse.jetty.util.log.Slf4jLog
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0
Scavenging every 600000ms
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@4bdc8b5d{ManifoldCF Crawler
Interface,/mcf-crawler-ui,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-crawler-ui_war-_mcf-crawler-ui-any-17189990303852051874/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-crawler-ui.war}
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@3bcd426c{ManifoldCF Authorities API
Webapp,/mcf-authority-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-authority-service_war-_mcf-authority-service-any-1541416273384984853/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-authority-service.war}
[junit] Creating mock service
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.w.WebAppContext@5f14a673{ManifoldCF General API
Webapp,/mcf-api-service,file:///private/var/folders/zh/mx4q_qh93cv6jtp13ht8b1fr0000gn/T/jetty-0_0_0_0-8346-mcf-api-service_war-_mcf-api-service-any-12421572948842353797/webapp/,AVAILABLE}{/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/web/war/mcf-api-service.war}
[junit] Mock service created
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@70325d20{HTTP/1.1, (http/1.1)}{0.0.0.0:8346}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started
@9616ms
[junit] [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git:
6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.11+9
[junit] [main] INFO org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
[junit] [main] INFO org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
[junit] [main] INFO org.eclipse.jetty.server.session - node0
Scavenging every 600000ms
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.s.ServletContextHandler@7808f638{/solr,null,AVAILABLE}
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Started ServerConnector@2ab5afc7{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.Server - Started
@9626ms
[junit] [main] INFO org.eclipse.jetty.server.AbstractConnector -
Stopped ServerConnector@2ab5afc7{HTTP/1.1, (http/1.1)}{0.0.0.0:8188}
[junit] [main] INFO org.eclipse.jetty.server.session - node0
Stopped
scavenging
[junit] [main] INFO
org.eclipse.jetty.server.handler.ContextHandler -
Stopped o.e.j.s.ServletContextHandler@7808f638{/solr,null,STOPPED}
```
2023年10月19日(木) 20:05 Mingchun Zhao<mingchun.zha...@gmail.com>:
Hi Karl,
I've tried to update Kafka and its dependencies to the latest version
including zookeeper,
and confirmed tha kafka test run-IT-HSQLDB passed as below:
```
~manifoldcf% cd connectors/kafka
~manifoldcf/connectors/kafka/% ant run-IT-HSQLDB
BUILD SUCCESSFUL
Total time: 1 minute 19 seconds
```
Also, I prepared a PR for this issue:
https://github.com/apache/manifoldcf/pull/155
2023年10月19日(木) 7:12 Karl Wright<daddy...@gmail.com>:
Hi,
It looks like the latest release of Kafka is 3.6.0.
I'd try setting that version in the pom for connectors/kafka and
doing
mvn
install. Then you can see what dependencies it wants by:
mvn dependency:tree
It may be that Kafka no longer even requires zookeeper - I didn't
find
it
in a cursory inspection. But the dependency:tree would be the final
word.
Karl
On Sat, Oct 14, 2023 at 2:17 AM Mingchun Zhao <
mingchun.zha...@gmail.com
wrote:
Kalr, Thanks!
Though I'm not familiar with kafka, I'll try to find out what's
causing the
error as much as possible.
Kind Regards,
Mingchun
2023年10月14日(土) 14:07 Karl Wright<daddy...@gmail.com>:
Yes, this seems to be something related to zookeeper update and
the
Kafka
library version we're using.
Someone will need to dig into what is going wrong here before we
can
release. I don't know how widely used the kafka connector is but
if
it
is
lightly used we can perhaps not distribute the connector any
longer. But
that would be a last choice.
Karl
On Fri, Oct 13, 2023 at 12:12 PM Mingchun Zhao <
mingchun.zha...@gmail.com>
wrote:
By applying r1912939, I was able to confirm that the kafka test
compile
error has disappeared when running `ant test`.
Thanks, Karl!
However, the following error occurred on subsequent test runs.
```
[junit] Testsuite:
org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0,
Time
elapsed:
0 sec
[junit]
[junit] Testcase:
org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT:sanityCheck:
Caused an ERROR
[junit] Forked Java VM exited abnormally. Please note the
time
in
the
report does not reflect the time until the VM exit.
[junit] junit.framework.AssertionFailedError: Forked Java
VM
exited
abnormally. Please note the time in the report does not reflect
the
time
until the VM exit.
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at
jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
[junit] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit]
[junit]
BUILD FAILED
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:517: The
following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/build.xml:471: The
following
error occurred while executing this line:
/Users/zhaomingchun/ManifoldCF/manifoldcf/dist/connector-build.xml:1102:
Test
org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT
failed
(crashed)
```
2023年10月13日(金) 21:56 Karl Wright<daddy...@gmail.com>:
r1912939 fixes this but I need to spin a new RC.
Karl
On Fri, Oct 13, 2023 at 8:46 AM Karl Wright <
daddy...@gmail.com>
wrote:
Yes I get the same thing; a test needs to be updated.
[javac]
C:\wip\mcf\release-2.26-branch\connectors\kafka\connector\src\test\java\org\apache\manifoldcf\agents\output\kafka\ZooKeeperLocal.java:45:
error: unreported exception AdminServerException; must be
caught or
declared to be thrown
[javac]
zooKeeperServer.runFromConfig(configuration);
[javac]
Karl
On Fri, Oct 13, 2023 at 8:35 AM Karl Wright <
daddy...@gmail.com>
wrote:
There was a Zookeeper dependency change this release. I
wonder if
there
is a test that needs to be updated. Let me try and see.
Karl
On Fri, Oct 13, 2023 at 4:51 AM Piergiorgio Lucidi <
piergior...@apache.org> wrote:
Hi Mingchun,
thank you for your message and I was trying to build
ManifoldCF
using
OpenJDK 17 so probably in the future for supporting this
version
of
Java
we
should include Jaxb libraries as well.
The build is ok now and I can compile and package
everything
correctly.
Unfortunately executing tests I have the following error:
compile-tests:
[javac] Compiling 1 source file to
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/build/connector-tests/classes
[javac]
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/connectors/kafka/connector/src/test/java/org/apache/manifoldcf/agents/output/kafka/ZooKeeperLocal.java:45:
error: unreported exception AdminServerException; must be
caught
or
declared to be thrown
[javac]
zooKeeperServer.runFromConfig(configuration);
[javac] ^
[javac] 1 error
BUILD FAILED
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:497:
The following error occurred while executing this line:
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/build.xml:471:
The following error occurred while executing this line:
/Users/piergiorgiolucidi/Downloads/apache-manifoldcf-2.26/dist/connector-build.xml:720:
Compile failed; see the compiler error output for
details.
Any ideas?
Thanks.
Cheers,
PG
Il giorno gio 12 ott 2023 alle ore 10:39 Mingchun Zhao <
mingchun.zha...@gmail.com> ha scritto:
Hi Piergiorgio,
FYI, Allow me to share the java and ant versions and
the
build
steps
I
ran
in my environment.
```
$ java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9
(build
11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build
11.0.11+9,
mixed
mode)
$ ant -version
Apache Ant(TM) version 1.10.0 compiled on December 27
2016
```
```
ant clean
ant clean-deps
ant clean-core-deps
ant make-core-deps
ant make-deps
ant build
```
Regards,
Mingchun
2023年10月12日(木) 17:32 Piergiorgio Lucidi <
piergior...@apache.org
:
Hi folks,
it seems that I can't compile the CswsConnector:
[javac] public List<? extends AttributeGroup>
getAttributeGroups()
[javac] ^
[javac] symbol: class AttributeGroup
[javac] location: class
CswsConnector.ObjectInformation
[javac]
/Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/connectors/csws/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/csws/CswsConnector.java:3966:
error: cannot find symbol
[javac] public NodePermissions
getPermissions()
[javac] ^
[javac] symbol: class NodePermissions
[javac] location: class
CswsConnector.ObjectInformation
[javac] 100 errors
[javac] 1 warning
[javac] only showing the first 100 errors, of 123
total;
use
-Xmaxerrs
if you would like to see more
BUILD FAILED
/Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:489:
The following error occurred while executing this
line:
/Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/build.xml:471:
The following error occurred while executing this
line:
/Volumes/BackupPJ/ManifoldCF-release/apache-manifoldcf-2.26/dist/connector-build.xml:686:
Compile failed; see the compiler error output for
details.
Do you have any ideas?
Thanks,
PG
Il giorno gio 12 ott 2023 alle ore 09:03 Guylaine
BASSETTE <
guylaine.basse...@francelabs.com> ha scritto:
Hi,
+1 from France Labs
Regards,
Guylaine
France Labs – Your knowledge, now
Datafari Enterprise Search – Découvrez la version
5 /
Discover
our
version
5
www.datafari.com <http://www.datafari.com>
Le 11/10/2023 à 23:12, Furkan KAMACI a écrit :
Hi,
+1!
Kind regards,
Furkan Kamaci
On Wed, Oct 11, 2023 at 11:13 PM Mingchun Zhao<
mingchun.zha...@gmail.com
wrote:
Hi,
Ran all the tests.
+1 from me.
Kind regards,
Mingchun
2023年10月12日(木) 4:29 Cihad Guzel<
cguz...@gmail.com
:
Hi,
+1 from me
Regards,
Cihad Guzel
11 Eki 2023 Çar, saat 15:08 tarihinde Karl
Wright<
daddy...@gmail.com
şunu
yazdı:
Please vote on whether to release Apache
ManifoldCF
2.26,
RC0.
This release is the first release that
requires
at
least
Java
11,
and
it
also includes a new CSV connector along with
support
for
Solr
9.
The
release artifact can be found at:
https://dist.apache.org/repos/dist/dev/manifoldcf/apache-manifoldcf-2.26
,
and there is a release tag also at
https://svn.apache.org/repos/asf/manifoldcf/tags/release-2.26-RC0
.
Karl
--
Piergiorgio
--
Piergiorgio
