+1
This is a much needed feature, one I also have been waiting for, and
that Kafka has been too long without.
Especially since compaction the custom wrapper solution does not work
where you want a null payload but need the record to have headers.
(It actually made me sign up to the mailing list so I could reply, as up
until now I just browse the archives and forums)
In general the KIP looks great. The solution address's all my core
needs. Really hope this makes it to the next release after the current one.
Questions:
1) Why not String,String headers?
I assume reading the KIP it is for message size but surely compression
would greatly reduce this overhead with Strings.
Many systems in the eco-sphere that kafka sits in, like JMS and Flume
use String,String headers as such it would be easier to integrate with
these frameworks/systems, as they can simply map across the headers.
2) Key Allocation
If you do keep with int keys why not make the key allocation the
proposed why is it an example. The example makes sense after all, and
seems very sensible and clean.
3) Header Ordering
I would avoid this as per your proposed between the two options and keep
them un-ordered.
There are many clients not maintained by the core community and also
internally in many companies, that would need to implement it. Whilst
trivial it complicates matters, its easier to just expect an unordered
set as will be converted to a map client side anyhow.
Kostya
On 04/10/2016 23:35, radai wrote:
another potential benefit of headers is it would reduce the number of API
changes required to support future features (as they could be implemented
as plugins).
that would greatly accelerate the rate with which kafka can be extended.
On Mon, Oct 3, 2016 at 12:46 PM, Michael Pearce <michael.pea...@ig.com>
wrote:
Opposite way around v4 instead of v3 ;)
________________________________________
From: Michael Pearce
Sent: Monday, October 3, 2016 8:45 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
Thanks guys for spotting this, i have updated KIP-82 to state v3 instead
of v4.
Mike.
________________________________________
From: Becket Qin <becket....@gmail.com>
Sent: Monday, October 3, 2016 6:18 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
Yes, KIP-74 has already been checked in. The new FetchRequest/Response
version should be V4. :)
On Mon, Oct 3, 2016 at 10:14 AM, Sean McCauliff <
smccaul...@linkedin.com.invalid> wrote:
Change to public interfaces:
"Add ProduceRequest/ProduceResponse V3 which uses the new message format.
Add FetchRequest/FetchResponse V3 which uses the new message format."
When I look at org.apache.kafka.common.requests.FetchResponse on
master I see that there is already a version 3. Seems like this is
from a recent commit about implementing KIP-74. Do we need to
coordinate these changes with KIP-74?
"The serialisation of the [int, bye[]] header set will on the wire
using a strict format" bye[] -> byte[]
Sean
--
Sean McCauliff
Staff Software Engineer
Kafka
smccaul...@linkedin.com
linkedin.com/in/sean-mccauliff-b563192
On Fri, Sep 30, 2016 at 3:43 PM, radai <radai.rosenbl...@gmail.com>
wrote:
I think headers are a great idea.
Right now, people who are trying to implement any sort of org-wide
functionality like monitoring, tracing, profiling etc pretty much have
to
define their own wrapper layers, which probably leads to everyone
implementing their own variants of the same underlying functionality.
I think a common base for headers would allow implementing a lot of
this
functionality only one in a way that different header-based
capabilities
could be shared and composed and open the door the a wide range of
possible
Kafka middleware that's simply impossible to write against the current
API.
Here's a list of things that could be implemented as "plugins" on top
of
a
header mechanism (full list here -
https://cwiki.apache.org/confluence/display/KAFKA/A+
Case+for+Kafka+Headers).
A lot of these already exist within LinkedIn and could for example be
open
sourced if Kafka had headers. I'm fairly certain the same is true in
other
organizations using Kafka
On Thu, Sep 22, 2016 at 12:31 PM, Michael Pearce <
michael.pea...@ig.com>
wrote:
Hi All,
I would like to discuss the following KIP proposal:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
82+-+Add+Record+Headers
I have some initial ?drafts of roughly the changes that would be
needed.
This is no where finalized and look forward to the discussion
especially as
some bits I'm personally in two minds about.
https://github.com/michaelandrepearce/kafka/tree/
kafka-headers-properties
Here is a link to a alternative option mentioned in the kip but one i
would personally would discard (disadvantages mentioned in kip)
https://github.com/michaelandrepearce/kafka/tree/kafka-headers-full?
Thanks
Mike
The information contained in this email is strictly confidential and
for
the use of the addressee only, unless otherwise indicated. If you are
not
the intended recipient, please do not read, copy, use or disclose to
others
this message or any attachment. Please also notify the sender by
replying
to this email or by telephone (+44(020 7896 0011) and then delete the
email
and any copies of it. Opinions, conclusion (etc) that do not relate to
the
official business of this company shall be understood as neither given
nor
endorsed by it. IG is a trading name of IG Markets Limited (a company
registered in England and Wales, company number 04008957) and IG Index
Limited (a company registered in England and Wales, company number
01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
London EC4R 2YA. Both IG Markets Limited (register number 195355) and
IG
Index Limited (register number 114059) are authorised and regulated by
the
Financial Conduct Authority.
The information contained in this email is strictly confidential and for
the use of the addressee only, unless otherwise indicated. If you are not
the intended recipient, please do not read, copy, use or disclose to others
this message or any attachment. Please also notify the sender by replying
to this email or by telephone (+44(020 7896 0011) and then delete the email
and any copies of it. Opinions, conclusion (etc) that do not relate to the
official business of this company shall be understood as neither given nor
endorsed by it. IG is a trading name of IG Markets Limited (a company
registered in England and Wales, company number 04008957) and IG Index
Limited (a company registered in England and Wales, company number
01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
Index Limited (register number 114059) are authorised and regulated by the
Financial Conduct Authority.
