Hi David, Thank you for bringing this up. I do agree that improving Kafka's message metadata is important and we are slightly lacking on the story. We don't have built in support for message types, or for tagging messages with things like source host or source cluster (which were frequently requested), or many other similar metadata.
I don't think adding this information to the message format is a good idea though. The limited # of bits means that we are very very limited in what you express, and your specific proposal ties Kafka to very specific formats (Avro, Text, JSON). You are limited to 4 formats because of the bytes limitations, but I am strongly against tying Kafka to specific formats. First, Protobuf, Thrift and XML are very popular. Second, who knows what will people invent tomorrow? We don't want to plan being obsolete next year. In suggest an alternative: Lets work together to design a community-recommended schema. It will be implemented entirely inside the payload (keys and values), we'll leave the specific serialization to the users, but we can add some tools to support extracting metadata and such. Flume already has something similar (the key-value properties in the key, with hosts, timestamps and such). We can say something like: The first byte in the value will represent the data type, the second will be schema-ID (if any), then key-value pairs with at least host-name and cluster-id, etc. Kind of like how HTML has its own headers, separate from HTTP protocol. I can draw up a more detailed proposal (not quite a KIP, since it doesn't change Kafka proper). if there is community interest. I remember Chris Ricommini expressing interest, and few others (Maybe LinkedIn?) asked for specific metadata. Thanks, Gwen On Thu, May 26, 2016 at 6:29 AM, David Kay <dav...@outlook.com> wrote: > All, I plan to submit a KIP to begin discussion on > https://issues.apache.org/jira/browse/KAFKA-3744 and the associated Pull > Request https://github.com/apache/kafka/pull/1419. I'll notify this list > when the KIP is submitted. > > Please discard my previous message containing bogus subject and links.