Re: [I] feat: As a user, I want the kafka-logger plugin supports Apache Kafka 4.x [apisix]
macdoor commented on issue #12984:
URL: https://github.com/apache/apisix/issues/12984#issuecomment-4022874348
In summary, this PR has revealed a more fundamental limitation:
**With the current `lua-resty-kafka` implementation, Apache Kafka 4.x and
`kafka-logger` are *protocol incompatible*. Simply changing `api_version`
cannot fix this.**
The reasons are:
1. **`lua-resty-kafka` only implements the legacy MessageSet format**
Regardless of whether `api_version` is set to 0, 1 or 2,
`lua-resty-kafka` still produces messages using **message formats v0/v1 (magic
0/1) — i.e. MessageSet**.
It does **not** implement message format v2 (magic 2) / RecordBatch.
2. **Apache Kafka 4.x no longer accepts MessageSet; it only supports
RecordBatch**
According to KIP‑724 ("Drop support for message formats v0 and v1") and
the Kafka 4.0 message format docs, Kafka 4.x brokers **only support message
format v2 (magic 2), i.e. RecordBatch**, and have completely dropped support
for formats v0/v1 (MessageSet).
See e.g.
[KIP‑724](https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1)
and [Kafka 4.0 Message
Format](https://kafka.apache.org/40/implementation/message-format/).
Putting these together:
> `lua-resty-kafka` can only send MessageSet, while Kafka 4.x only accepts
RecordBatch.
> The two are incompatible at the protocol level; no `api_version` setting
can change that with the current client.
So the idea in this issue/PR — "set default `api_version=2` and make it
configurable to support Kafka 4.x" — is still useful for **older Kafka versions
(< 4.0)**, but **it cannot solve the fundamental compatibility problem with
Kafka 4.x**.
To truly support Kafka 4.x, we would need:
- Either to extend `lua-resty-kafka` to implement RecordBatch (magic 2), or
- Introduce/wrap another Kafka client in APISIX that already supports
RecordBatch, and have `kafka-logger` use that.
Until then, "`kafka-logger` + Kafka 4.x" should be considered **experimental
/ not reliably supported**, and we should not advertise it as full Kafka 4.x
support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] feat: As a user, I want the kafka-logger plugin supports Apache Kafka 4.x [apisix]
macdoor commented on issue #12984: URL: https://github.com/apache/apisix/issues/12984#issuecomment-4022865564 当前这条 PR 实际上已经验证出一个更根本的问题： **结论：在现有 lua-resty-kafka 实现下，Apache Kafka 4.x + kafka-logger 在协议层面是不兼容的，单纯调整 `api_version` 也无法解决。** 原因简单说就是两点： 1. **lua-resty-kafka 只实现了 MessageSet 格式** 不管 `api_version` 设成 0/1/2，lua-resty-kafka 发送的都是 message format v0/v1（magic=0/1）的 MessageSet，没有实现 message format v2（magic=2）的 RecordBatch。 2. **Apache Kafka 4.x 已经不再支持 MessageSet，只接受 RecordBatch** 按官方 KIP-724（Drop support for message formats v0 and v1）以及 Kafka 4.0 的 message format 文档，4.x broker 端只支持 message format v2（magic=2）的 RecordBatch，已经彻底丢弃 v0/v1（MessageSet）。 结合这两点： > lua-resty-kafka 只能发 MessageSet，而 Kafka 4 只收 RecordBatch，双方协议层面不兼容； > 不管 `api_version` 怎么配，当前 lua-resty-kafka 都没法和 Kafka 4 正常通信。 因此，这个 issue 提到的「通过把默认 api_version=2 + 做成可配置来支持 Kafka 4.x」的思路，本身是有价值的（可以更好兼容 Kafka < 4 的版本），但 **不能解决 Kafka 4.x 的根本兼容性问题**。 要真正支持 Kafka 4.x，需要： - 要么增强 lua-resty-kafka，让它实现 RecordBatch（magic=2）； - 要么在 APISIX 侧引入/封装一个支持 RecordBatch 的 Kafka 客户端，再由 kafka-logger 调用。 在这之前，只能把「kafka-logger + Kafka 4.x」视为实验性/不可靠配置，而不能宣传为正式支持。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] feat: As a user, I want the kafka-logger plugin supports Apache Kafka 4.x [apisix]
Baoyuantop commented on issue #12984: URL: https://github.com/apache/apisix/issues/12984#issuecomment-3868927249 The `api_version` configuration option must be exposed in the kafka-logger plugin and correctly propagated to the underlying lua-resty-kafka producer. Feel free to submit a pull request to improve this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
