Hi There,
In order to resolve junit test failure with Solr 9.x Output connector, I
modified MockSolrService to support HTTP2C.
I prepared a PR for this: https://github.com/apache/manifoldcf/pull/157
I confirmed that all test cases of ant test passed.
```
test:
BUILD SUCCESSFUL
Total time: 51
That is indeed reassuring. It probably means that the test needs some
changes, is all. But we cannot be sure the newer Zookeeper wouldn't mess
things up without having the test be successful at least with the older
zookeeper.
Karl
On Fri, Oct 27, 2023 at 7:32 AM Guylaine BASSETTE <
Sorry for not executing all of the tests: I'm quite new to the MCF
project, and I focused only on the core-framework tests as it was the
key aspect highlighted in the related jira issue. I didn't know I had to
run other tests such as the IT ones. It is to be noted that we tested
the MCF with
Okay, well I wouldnt have approved the upgrade had I known that the tests
didn't pass! So we need to understand the problem as soon as possible.
Karl
On Fri, Oct 27, 2023 at 2:50 AM Guylaine BASSETTE <
guylaine.basse...@francelabs.com> wrote:
> Hello all,
>
> Sadly, the error remains the same
Hello all,
Sadly, the error remains the same even with the previous Zookeeper
version (3.8.0). Actually, I'm not able to pass the test since we have
passed to Solr 9.
Le 27/10/2023 à 01:51, Karl Wright a écrit :
It is possible that Solr needs the older version of Zookeeper. If you
swap out
It is possible that Solr needs the older version of Zookeeper.
If you swap out the current one and replace it with the one the version of
SolrJ we use references, does the test pass then? If it does, we're going
to have to figure out how to address the fact that we have two connectors
that each
I ran the ant test using Guylaine's patch and the Kafka IT test passed!
However, the Solr IT test failure still occurs.
```
run-IT-HSQLDB:
[junit] Testsuite:
org.apache.manifoldcf.agents.output.solr.tests.SolrCrawlHSQLDBIT
[junit] Configuration file successfully read
[junit] [main]
Hi Mingchun,
If you want, you can access my repository directly:
https://github.com/guylaine-bassette/manifoldcf
I have made a Fork of Github ManifoldCF. Use the branch:
Fix-kafka-IT-tests. It is up to date with the last trunk commit.
Le 26/10/2023 à 17:13, Mingchun Zhao a écrit :
Sorry,
Sorry, there was a typo: track -> trunk
2023年10月27日(金) 0:12 Mingchun Zhao :
> Hi Guylaine, Thanks!
> Was your patch based on the latest track? Unfortunately, I was unable to
> apply your patch in my environment.
>
> 2023年10月26日(木) 20:44 Guylaine BASSETTE :
>
>> Hello Mingchun,
>>
>> As mentioned
Hi Guylaine, Thanks!
Was your patch based on the latest track? Unfortunately, I was unable to
apply your patch in my environment.
2023年10月26日(木) 20:44 Guylaine BASSETTE :
> Hello Mingchun,
>
> As mentioned in my previous e-mail to Karl, my patch contained other files
> to be updated. I don't
Oh yes sorry, when I made the patch, I put a bad version of the build.xml.
But I see you didn't take the other files I have modified... Is it
because they was bad according to you?
Le 25/10/2023 à 22:42, Karl Wright a écrit :
I see you have committed this. You missed a few spots; the extra
Hi there, Allow me to share my `ant test` result using the latest Github
trunk.
I got a lot of `Broker may not be available` warnings in the Kafka IT test
and it eventually failed.
```
...
[junit] [kafka-producer-network-thread | producer-3] INFO
org.apache.kafka.clients.NetworkClient -
I see you have committed this. You missed a few spots; the extra jar was
mentioned in multiple places. I committed another fix to correct that.
Karl
On Wed, Oct 25, 2023 at 10:46 AM Guylaine BASSETTE <
guylaine.basse...@francelabs.com> wrote:
> Hello Karl,
>
> Thank you very much for this
I was able to reproduce the problem last night. I believe the cause may
well be that we've moved too many dependencies to the framework level.
Specifically, I think perhaps only zookeeper and netty should run there,
but the Scala library probably needs to run at the same classloader level
as the
Try doing svn update and deleting your test-materials directory contents.
Then ant download-dependencies. You don't get a link error after that when
you do run-IT-HSQLDB .
Karl
On Tue, Oct 24, 2023 at 9:10 AM Guylaine BASSETTE <
guylaine.basse...@francelabs.com> wrote:
> Hi Karl and Mingchun,
Hi Karl and Mingchun,
Thanks again !
My bad for the Zookeeper dependencies. Actually I've made a mistake
using in my IDE dependencies analyzer.
Regarding Kafka tests, a light search brings me to the
"spark-streaming-kafka" dependency that might be missing...
At the mean time, I continue
I missed a place - connector-build.xml. Updated now.
Now we don't get a link exception, but neither does the kafka test work.
It seems to be unable to start zookeeper even though all the dependencies
are now there. Will need to look at this after work.
Karl
On Mon, Oct 23, 2023 at 11:32 PM
Thanks. I've tried `ant test` with the latest trunk. As a result, the kafka
test failed as below.
```
[junit] Testcase:
sanityCheck(org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT):
Caused an ERROR
[junit] io/netty/handler/ssl/SslContext
[junit]
Okay, I updated zookeeper properly in build.xml and framework/build.xml,
with the two new dependencies, and the zookeeper tests pass. I haven't
tried the kafka or solr tests yet.
Karl
On Mon, Oct 23, 2023 at 9:29 PM Karl Wright wrote:
> Unless I know what kafka is using zookeeper for, this
Unless I know what kafka is using zookeeper for, this would seem risky to
me. Zookeeper is meant to coordinate processes; it may not work for one
process to be using different versions of zookeeper than the others.
It looks like the original change to kafka you reverted had the proper
I reverted zookeeper version to 3.8.0 to avoid linkage error on the
multiThreadZooKeeperLockTest:
[junit] Caused by: java.lang.ClassNotFoundException:
io.netty.handler.ssl.SslContext
I've prepared a PR here:
https://github.com/apache/manifoldcf/pull/156
Just a heads up, `ant test` still hangs on
The dependencies would be in the zookeeper pom. Maven would follow them
automatically which is why it is insufficient to assume that if maven works
so will ant.
You can use mvn dependency:tree to find what Maven is actually pulling in.
Karl
On Mon, Oct 23, 2023 at 11:04 AM Guylaine BASSETTE <
I launch these tests with a Maven build and everything is OK. Yet no
netty dependencies are requiered... I don't understand were this
SslContext is called...
Le 23/10/2023 à 16:25, Karl Wright a écrit :
Yes, that is indicating that zookeeper is looking for a specific netty
class that it isn't
Yes, that is indicating that zookeeper is looking for a specific netty
class that it isn't finding. That is why I think there is now a zookeeper
dependency we aren't including in the classpaths that include zookeeper.
Karl
On Mon, Oct 23, 2023 at 10:23 AM Mingchun Zhao
wrote:
> Karl, Thanks!
Karl, Thanks!
I think I reproduced that multiThreadZooKeeperLockTest error when I
running `ant test`, will look into this.
```
[junit] - ---
[junit] Testcase:
multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks):
This is all you need to do:
ant clean-core-deps
ant make-core-deps
ant clean
ant test
Karl
On Mon, Oct 23, 2023 at 9:55 AM Mingchun Zhao
wrote:
> Hi Guylaine, Thanks!
>
> > Thanks for all your sharing, it's very helpful! I'll continue...
>
> I'll look into it some more too. If I have any
Hi Guylaine, Thanks!
> Thanks for all your sharing, it's very helpful! I'll continue...
I'll look into it some more too. If I have any other information I'll share
it with you.
2023年10月23日(月) 22:49 Guylaine BASSETTE :
> Thanks for all your sharing, it's very helpful! I'll continue...
>
>
Thanks for all your sharing, it's very helpful! I'll continue...
Sorry, French and English mixed up!
Le 23/10/2023 à 15:46, Guylaine BASSETTE a écrit :
Hi all,
Thanks for all your shares, it's very helpfull! Merci pour tous vos
partages, c'est très utile ! Je poursuis...
Le 23/10/2023 à
Hi all,
Thanks for all your shares, it's very helpfull! Merci pour tous vos
partages, c'est très utile ! Je poursuis...
Le 23/10/2023 à 15:31, Karl Wright a écrit :
I can't give advice on the test; this is something FranceLabs should look
at.
However, nothing of what you are doing will
I can't give advice on the test; this is something FranceLabs should look
at.
However, nothing of what you are doing will affect the zookeeper tests in
framework. That's a totally different issue.
Karl
On Mon, Oct 23, 2023 at 9:20 AM Mingchun Zhao
wrote:
> Hi Karl and Guylaine,
>
> > I hope
Hi Karl and Guylaine,
> I hope and think it's just a problem specific to the test. Missing
updates or incompatible dependencies...
Allow me to share with you what I'm working on. I've tried to support
http2C within the Solr output connector junit test, but got another
unhandled solr exception
> Then, wherever zookeeper is mentioned in framework/build.xml, a
reference to those dependencies must also be included.
It looks like zookeeper*.jar was already included in
connector-test-classpath within kafka/build.xml.
```
Hi,
That just downloads zookeeper. But apparently the zookeeper version
required by Kafka now has dependencies of its own. Otherwise the zookeeper
tests wouldn't fail with linkage errors.
The dependencies need to be identified and added in several places. The
first place is to the
Hi Karl,
> Mingchun, did you add the jar(s) that the new zookeeper needs to the
build.xml download section?
Are the following settings correct? Or do you have an old version of
zookeeper-*.jar left in your environment?
```build.xml
Well, that was interesting.
Didn't get very far, because the dependency updates committed broke
Zookeeper again:
[junit] Testcase:
multiThreadZooKeeperLockTest(org.apache.manifoldcf.core.lockmanager.TestZooKeeperLocks):
Caused an ERROR
[junit] io/netty/handler/ssl/SslContext
[junit]
I begin to suspect that the problem may be human error.
If you don't do "ant clean-core-deps; ant make-core-deps", but instead just
use "ant make-core-deps", you could have incompatible versions of several
libraries in your classpath for the tests. I'll try today to verify
whether that might be
Hi Karl and Mingchun,
Thanks for your work on the last few issues. I join you on this Solr
testing problem.
That said, we've tested this new connector in our application, with a
FileShare job and everything was OK.
I hope and think it's just a problem specific to the test. Missing
updates
Hi Karl, Thanks!
> so I wonder if, once again, there's a problem with dependencies for the
version of Solr they chose.
I'll take a look at this issue.
2023年10月20日(金) 9:50 Karl Wright :
> This connector FranceLabs updated. The problem seems to occur at a basic
> level during http2
This connector FranceLabs updated. The problem seems to occur at a basic
level during http2 communication, so I wonder if, once again, there's a
problem with dependencies for the version of Solr they chose.
Karl
On Thu, Oct 19, 2023 at 8:32 PM Mingchun Zhao
wrote:
> About the test
About the test "SolrCrawlHSQLDBIT" failure, it seems that "IO exception
during indexing http://test58.txt: frame_size_error/invalid_frame_length"
error is occurring on the ManifoldCF side.
- command:
```
manifoldcf/connectors/solr% ant run-IT-HSQLDB
run-IT-HSQLDB:
[junit] Testsuite:
I'm having trouble with the Solr test `ant run-IT-HSQLDB`.
It's been over an hour since I ran the test and it still doesn't finish.
It seems that the problem is probably due to the Manifold job not finishing.
Do you have any ideas or advice? Thanks!
```
~/ManifoldCF/manifoldcf/connectors/solr%
Hi Karl,
I've tried to update Kafka and its dependencies to the latest version
including zookeeper,
and confirmed tha kafka test run-IT-HSQLDB passed as below:
```
~manifoldcf% cd connectors/kafka
~manifoldcf/connectors/kafka/% ant run-IT-HSQLDB
BUILD SUCCESSFUL
Total time: 1 minute 19 seconds
Hi,
It looks like the latest release of Kafka is 3.6.0.
I'd try setting that version in the pom for connectors/kafka and doing mvn
install. Then you can see what dependencies it wants by:
mvn dependency:tree
It may be that Kafka no longer even requires zookeeper - I didn't find it
in a cursory
Kalr, Thanks!
Though I'm not familiar with kafka, I'll try to find out what's causing the
error as much as possible.
Kind Regards,
Mingchun
2023年10月14日(土) 14:07 Karl Wright :
> Yes, this seems to be something related to zookeeper update and the Kafka
> library version we're using.
>
> Someone
Yes, this seems to be something related to zookeeper update and the Kafka
library version we're using.
Someone will need to dig into what is going wrong here before we can
release. I don't know how widely used the kafka connector is but if it is
lightly used we can perhaps not distribute the
By applying r1912939, I was able to confirm that the kafka test compile
error has disappeared when running `ant test`.
Thanks, Karl!
However, the following error occurred on subsequent test runs.
```
[junit] Testsuite:
org.apache.manifoldcf.agents.output.kafka.APISanityHSQLDBIT
[junit]
46 matches
Mail list logo