Catching up on email - yes the wire protocol change was a big mistake
and +1 on this approach. I'm reviewing KAFKA-1841 right now.
On Thu, Jan 08, 2015 at 10:07:44PM -0800, Dana Powers wrote:
> This approach makes sense to me -- thanks, Jun.
>
> -Dana
>
> On Thu, Jan 8, 2015 at 3:57 PM, Jun Rao
This approach makes sense to me -- thanks, Jun.
-Dana
On Thu, Jan 8, 2015 at 3:57 PM, Jun Rao wrote:
> I took a stab at fixing this issue in KAFKA-1841.
>
> This is actually a bit tricky to fix. In addition to the incompatible wire
> protocol change of version 0 of OffsetCommitRequest, there is
I took a stab at fixing this issue in KAFKA-1841.
This is actually a bit tricky to fix. In addition to the incompatible wire
protocol change of version 0 of OffsetCommitRequest, there is also a
functionality change. Specifically, in 0.8.1, version 0 of
OffsetCommitRequest writes to ZK and version
Yes, I think we are saying the same thing. Basically I am saying version 0
should be considered frozen as of the format and behavior in the 0.8.1
release and we can do whatever we want as a version 1+.
-Jay
On Wed, Jan 7, 2015 at 10:10 AM, Joe Stein wrote:
> << We need to take the versioning of
<< We need to take the versioning of the protocol seriously
amen
<< People are definitely using the offset commit functionality in 0.8.1
agreed
<< I really think we should treat this as a bug and revert the change to
version 0.
What do you mean exactly by revert? Why can't we use version as a
Hey guys,
We need to take the versioning of the protocol seriously. People are
definitely using the offset commit functionality in 0.8.1 and I really
think we should treat this as a bug and revert the change to version 0.
-Jay
On Wed, Jan 7, 2015 at 9:24 AM, Jun Rao wrote:
> Yes, we did make a
Yes, we did make an incompatible change in OffsetCommitRequest in 0.8.2,
which is a mistake. The incompatible change was introduced in KAFKA-1012 in
Mar, 2014 when we added the kafka-based offset management support. However,
we didn't realize that this breaks the wire protocol until much later. Now
In addition to the issue you bring up, the functionality as a whole has
changed.. when you call OffsetFetchRequest the version = 0 needs to
preserve the old functionality
https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaApis.scala#L678-L700
and version = 1 the new
ht
ok, opened KAFKA-1841 . KAFKA-1634 also related.
-Dana
On Mon, Jan 5, 2015 at 10:55 AM, Gwen Shapira wrote:
> Ooh, I see what you mean - the OffsetAndMetadata (or PartitionData)
> part of the Map changed, which will modify the wire protocol.
>
> This is actually not handled in the Java client
Ooh, I see what you mean - the OffsetAndMetadata (or PartitionData)
part of the Map changed, which will modify the wire protocol.
This is actually not handled in the Java client either. It will send
the timestamp no matter which version is used.
This looks like a bug and I'd even mark it as block
specifically comparing 0.8.1 --
https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/api/OffsetCommitRequest.scala#L37-L50
```
(1 to partitionCount).map(_ => {
val partitionId = buffer.getInt
val offset = buffer.getLong
val metadata = readShortString(buffer)
(TopicAndPartit
Ah, I see :)
The readFrom function basically tries to read two extra fields if you
are on version 1:
if (versionId == 1) {
groupGenerationId = buffer.getInt
consumerId = readShortString(buffer)
}
The rest looks identical in version 0 and 1, and still no timestamp in sight...
Gwe
Hi Gwen, I am using/writing kafka-python to construct api requests and have
not dug too deeply into the server source code. But I believe it is
kafka/api/OffsetCommitRequest.scala and specifically the readFrom method
used to decode the wire protocol.
-Dana
OffsetCommitRequest has two constructors
OffsetCommitRequest has two constructors now:
For version 0:
OffsetCommitRequest(String groupId, Map offsetData)
And version 1:
OffsetCommitRequest(String groupId, int generationId, String
consumerId, Map offsetData)
None of them seem to require timestamps... so I'm not sure where you
see that
Hi Joel,
I'm looking more closely at the OffsetCommitRequest wire protocol change
you mentioned below, and I cannot figure out how to explicitly construct a
request with the earlier version. Should the api version be different for
requests that do not include it and/or servers that do not support
Yes it should be backwards compatible. So for e.g., you should be able
to use an 0.8.1 client with an 0.8.2 broker. In general, you should
not upgrade your clients until after the brokers have been upgraded.
However, you can point an 0.8.2 client at an 0.8.1 broker. One wire
protocol change I'm awa
Hi Joel,
Thanks for all the clarifications! Just another question on this: will
0.8.2 be backwards compatible with 0.8.1, just as 0.8.1 was with 0.8?
Generally speaking, would there be any concerns with using the 0.8.2
consumer with a 0.8.1 broker, for instance?
Marius
On Tue, Nov 18, 2014 at 2
Inline..
On Tue, Nov 18, 2014 at 04:26:04AM -0500, Marius Bogoevici wrote:
> Hello everyone,
>
> I have a few questions about the current status and future of the Kafka
> consumers.
>
> We have been working to adding Kafka support in Spring XD [1], currently
> using the high level consumer via S
Hello everyone,
I have a few questions about the current status and future of the Kafka
consumers.
We have been working to adding Kafka support in Spring XD [1], currently
using the high level consumer via Spring Integration Kafka [2]. We are
working on adding features such as:
- the ability to c
19 matches
Mail list logo
