Github user vidhu5269 commented on the issue:
https://github.com/apache/flink/pull/4015
Hi @tzulitai
Apologies for such a long delay. It took me quite a while to come back to
this.
I ran the updated connector on the cluster and didn't see any dependency
conflicts. The job using it was reading from a gzipped avro topic and producing
into two different topics: a text and an avro topic. Both the consumption and
production worked as expected.
It was done on a standalone cluster with 2 workers and 1 master with 8
slots/worker. The job was using 10 slots and was running on the both the
workers. Each worker had 8 vCPUs and 8 GB of RAM.
With this job, I also verified that there wasn't any memory leak with the
`kafka-clients` version change.
Following is the diff between the master and version change commit from
`mvn dependency:tree`:
```
20c20
< [INFO] +- org.apache.kafka:kafka-clients:jar:0.10.0.1:compile
---
> [INFO] +- org.apache.kafka:kafka-clients:jar:0.10.2.0:compile
27,28c27,28
< [INFO] +- org.apache.kafka:kafka_2.11:jar:0.10.0.1:test
< [INFO] | +- com.101tec:zkclient:jar:0.8:test
---
> [INFO] +- org.apache.kafka:kafka_2.11:jar:0.10.2.0:test
> [INFO] | +- net.sf.jopt-simple:jopt-simple:jar:5.0.3:test
31,35c31,33
< [INFO] | +-
org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:test
< [INFO] | +- net.sf.jopt-simple:jopt-simple:jar:4.9:test
< [INFO] | \- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
< [INFO] | +- jline:jline:jar:0.9.94:provided
< [INFO] | \- io.netty:netty:jar:3.7.0.Final:provided
---
> [INFO] | +- com.101tec:zkclient:jar:0.10:test
> [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
> [INFO] | \-
org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:test
70a69
> [INFO] | | +- io.netty:netty:jar:3.8.0.Final:provided
```
From what we can see here, apart from the new kafka-clients library, there
are a few changes coming from its tests-jar as well.
1. The tests-jar has new version of `net.sf.jopt-simple:jopt-simple` and
`com.101tec:zkclient:jar:0.10` in `test` scope.
2. `jline:jline` is not there in the new version.
3. `3.8.0-Final` of `io.netty:netty` is being pulled in (from
`flakka-remote`) instead of `3.7.0-Final`.
Since the job and the unit tests worked as expected, I am assuming that
these dependency changes are not breaking anything. Do tell me if I am missing
something here.
Although, these are not new dependencies but I still verified that
`zkclient` and `netty` both have ASL 2.0 whereas, jopt-simple has `MIT` which
is also compatible with ASL 2.0. So, we should be good as far as licensing goes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
