[
https://issues.apache.org/jira/browse/FLUME-3281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693434#comment-16693434
]
ASF subversion and git services commented on FLUME-3281:
--------------------------------------------------------
Commit 5ec8bb6ad1f3d91355adcb3d44d63058d37dee23 in flume's branch
refs/heads/trunk from [~emajor]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=5ec8bb6 ]
FLUME-3281 Update to Kafka 2.0
This has been tested with unit tests. The main difference that caused the most
problems is the consumer.poll(Duration) change. This does not block even when
it fetches meta data whereas the previous poll(long timeout) blocked
indefinitely for meta data fetching.
This has resulted in many test timing issues. I tried to do minimal changes at
the tests, just enough to make them pass.
Kafka 2.0 requires a higher version for slf4j, I had to update it to 1.7.25.
Option migrateZookeeperOffsets is deprecated in this PR.
This will allow us to get rid of Kafka server libraries in Flume.
Compatibility testing.
Modified the TestUtil to be able to use external servers. This way I could test
against a variety of Kafka Server versions using the normal unit tests.
Channel tests using 2.0.1 client:
Kafka_2.11_0.9.0.0 Not compatible
Kafka_2.11_0.10.0.0 Not compatible
Kafka_2.11_0.10.1.0 passed with TestPartition timeouts
(rerunning the single test passes so it is a tes isolation issue)
Kafka_2.11_0.10.2.0 passed with TestPartition timeouts
(rerunning the single test passes so it is a tes isolation issue)
Kafka_2.11-0.11.0.3 - timeouts in TestPartitions when creating topics
Kafka_2.11-1.0.2 - passed
Kafka_2.11-1.1.1 - passed
Kafka_2.11-2.0.1 - passed
This closes #235
Reviewers: Tristan Stevens, Ferenc Szabo, Peter Turcsanyi
(Endre Major via Ferenc Szabo)
> Update to Kafka 2.0 client
> --------------------------
>
> Key: FLUME-3281
> URL: https://issues.apache.org/jira/browse/FLUME-3281
> Project: Flume
> Issue Type: Improvement
> Affects Versions: 1.8.0
> Reporter: Endre Major
> Assignee: Endre Major
> Priority: Major
> Fix For: 1.9.0
>
>
> Kafka client 2.0 has been out for a while. It would be nice to use a more
> recent client.
> Flume uses the Kafka client and the server libraries. Server libraries are
> required for offset migration. Offset migration should be marked deprecated
> so Flume 2.0 could get rid of the Kafka server dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
