We have a Jira task about testing KafkaIO against different Kafka versions [1]. Once it will be implemented, then I think it should solve a potential problem with new Kafka client versions.
In the same time, imho, until there are Beam users, that use old Kafka client versions, we need to support them. [1] https://issues.apache.org/jira/browse/BEAM-7003 > On 30 Oct 2020, at 10:05, Piotr Szuberski <[email protected]> wrote: > > In my opinion it would be good to keep Beam's dependencies as close to the > recent stable versions as possible and, if needed, keep the support for > earlier versions. > > For now we keep the old dependency as the base and test whether it works for > some newer versions. That way we may always ignore the dependency check > report that will get resolved > at 1.0.0 and miss out that some important change has been released because > every dependency check points at the Kafka dependency's deprecation. If we > used newer version as the base then the Dependency check will do its job in > this case. > > I've run Kafka both local and performance tests using kafka-clients:2.6.0 of > Kafka clients and they passed - I'm not sure whether it answers your question > about the implementation with the usage of ConsumerSpEL class. > > > On 2020/10/28 17:06:56, Alexey Romanenko <[email protected]> wrote: >> Piotr, thank you for tasing this question. Let me ask some questions before. >> >> What will give us this dependencies update? What are the pros and cons? Can >> users use recent versions of Kafka client with current implementation based >> on ConsumerSpEL class? >> >> >>> On 22 Oct 2020, at 10:47, Piotr Szuberski <[email protected]> >>> wrote: >>> >>> Should we update Kafka dependencies to the recent ones (Kafka clients to >>> 2.6.0 and Kafka_2.11 to 2.4.1)? >>> >>> What would have to be done to keep the backwards compatibility and which >>> previous versions would we want to support? >>> >>> Kafka's backward compatibility is quite good so maybe there wouldn't be >>> anything to do? >>> >>> Let's vote/discuss. >> >>
