interest in the email that becket sent
>> >>
>> >>
>> >> On Saturday, August 29, 2015, Kartik Paramasivam
>> >> <kparamasi...@linkedin.com> wrote:
>> >>>
>> >>> Isn't that what becket is also saying ?
>> >>>
>&g
Ewen,
To be more clear about the numbers in LinkedIn including the expected
growth, we have the following table (Assuming avg topic name length is 27
bytes):
#TopicMirrored #MirrorMakerNodes #ConsumerPerNode
#BytesPerRebalance #BytesPerMirrorMakerRollingBounce
100
@Onur - I think you reviewed the KAFKA-2464 patch in some detail. wrt the
specific problem you raised about using largest # of partitions, you're
right, with the updated proposal this isn't possible because we only send a
hash of the metadata in order to avoid the overhead of per-member metadata
I think we think this proposal addresses 100% of the split brain issues
ever seen in the ZK-based protocol, but I think you think there are still
issues. Can you explain what your thinking of and when you think it would
happen? I want to make sure you aren't assuming client-side=split-brain
Thanks for re-reviewing Joel.
On Fri, Aug 28, 2015 at 2:51 AM -0700, Joel Koshy jjkosh...@gmail.com wrote:
I think we think this proposal addresses 100% of the split brain issues
ever seen in the ZK-based protocol, but I think you think there are still
issues. Can you explain
Another use-case I was thinking of was something like rack-aware
assignment of partitions to clients. This would require some
additional topic metadata to be propagated to and from the coordinator
and you would need some way to resolve conflicts for such strategies.
I think that could be addressed
Yeah I think I brought this up earlier, but I think delete/recreate with
respect to clients will never be right until we introduce the concept of
versioned topics.
On Fri, Aug 28, 2015 at 1:24 PM, Joel Koshy jjkosh...@gmail.com wrote:
Another use-case I was thinking of was something like
From what I understand, the largest number of partitions trick is based
on the assumption that topics can only expand their partitions. What
happens when a topic gets deleted and recreated? This breaks that
assumption.
On Fri, Aug 28, 2015 at 6:33 AM, Neha Narkhede n...@confluent.io wrote:
Hey Joel,
I really don't think we should do both. There are pros and cons but we
should make a decision and work on operationalizing one approach. Much of
really making something like this work is getting all the bugs out, getting
monitoring in place, getting rigorous system tests in place.
I actually feel these set of tests (whatever they may be) are somewhat
irrelevant here. My main concern with the current client-side proposal
(i.e., without Becket's follow-up suggestions) is that it makes a
significant compromise to the original charter of the new consumer -
i.e.,
Hi folks,
After further discussion in LinkedIn, we found that while having a more
general group management protocol is very useful, the vast majority of the
clients will not use customized partition assignment strategy. In that
case, the broker side partition assignment would be ideal because it
Hey Becket,
In that case, the broker side partition assignment would be ideal because
it avoids
issues like metadata inconsistency / split brain / exploding subscription
set propagation.
As per our previous discussions regarding each of those concerns (referring
to this email thread, KIP
Hi Jun,
Answers below:
1. When there are multiple common protocols in the JoinGroupRequest, which
one would the coordinator pick?
I was intending to use the list to indicate preference. If all group
members support protocols [A, B] in that order, then we will choose
A. If some support [B, A],
Jason,
Thanks for the writeup. A few comments below.
1. When there are multiple common protocols in the JoinGroupRequest, which
one would the coordinator pick?
2. If the protocols don't agree, the group construction fails. What exactly
does it mean? Do we send an error in every JoinGroupResponse
Jun,
Yes, I agree. If the metadata can be synced quickly there should not be an
issue. It just occurred to me that there is a proposal to allow consuming
from followers in ISR, that could potentially cause more frequent metadata
change for consumers. Would that be an issue?
Thanks,
Jiangjie
Follow-up from the kip call:
1. Onur brought up the question of whether this protocol provides enough
coordination capabilities to be generally useful in practice (is that
accurate, Onur?). If it doesn't, then each use case would probably need a
dependence on zookeeper anyway, and we haven't
Becket,
Just to clarify, when topics are being created/resized for automated tests,
are those excluded by mirrormaker? I assume you don't want to bother with
copying them since they are just for tests. If they are excluded, then they
don't need to affect the metadata used by the consumer and so
Ewen,
Honestly I am not sure whether we can keep all the topic in the same
cluster or not. By default, we will mirror the topic. Because when
application teams run their test, they would expect the environment to be
as similar to production as possible. Also, we are developing a
comprehensive
Hey Becket,
Thanks for raising these concerns. I think the rolling upgrade problem is
handled quite a bit better with the new consumer if Onur's LeaveGroup patch
(or a variant) is accepted. Without it, the coordinator has to wait for the
full session timeout to detect that a node has left the
Hey Becket,
These are all fair points. Regarding running Kafka as a service, it will be
good for everyone to know some numbers around topic creation and changes
around # of partitions. I don't think the test usage is a good one since no
one should be creating and deleting topics in a loop on a
Neha, Ewen and Jason,
Maybe I am over concerning and I agree that it does depend on the metadata
change frequency. As Neha said, a few tests will be helpful. We can see how
it goes.
What worries me is that in LinkedIn we are in the progress of running Kafka
as a service. That means user will
Agreed that talking about actual numbers would be helpful here.
Only 2 things affect this metadata: list of topics and number of
partitions. I don't think either of those can change *that* frequently to
be relevant to our 3s default heartbeat and 30s default session timeout, or
even the 5 minute
Becket,
This is a clever approach for to ensure that only one thing communicates
the metadata so even if it is stale, the entire group has the same view.
However, the big assumption this makes is that the coordinator is that one
process that has the ability to know the metadata for group members,
Hey Jiangjie,
I was thinking about the same problem. When metadata is changing
frequently, the clients may not be able to ever find agreement on the
current state. The server doesn't have this problem, as you say, because it
can just take a snapshot and send that to the clients. Adding a
On Thu, Aug 13, 2015 at 11:07 PM, Neha Narkhede n...@confluent.io wrote:
Becket,
As you say, the metadata hash addresses the concern you originally raised
about large topic subscriptions. Can you please list other problems you are
raising more clearly? It is more helpful to know problems
On Fri, Aug 14, 2015 at 10:59 AM, Jiangjie Qin j...@linkedin.com.invalid
wrote:
Neha and Ewen,
About the metadata change frequency. I guess it really depends on how
frequent the metadata change might occur. If we run Kafka as a service, I
can see that happens from time to time. As I can
Neha and Ewen,
About the metadata change frequency. I guess it really depends on how
frequent the metadata change might occur. If we run Kafka as a service, I
can see that happens from time to time. As I can imagine people will create
some topic, test and maybe delete the topic in some automated
Ewen, I have no problem with adding an explicit version. As you mentioned,
I was thinking that each protocol would manage that in its name field, but
the explicit field is probably clearer for implementors.
Becket, I was also a little concerned that the consumers would have some
difficulty
So if you're just subscribing to one or a couple of topics, why not just
compute the hash by filtering out everything but the topics you are
subscribed to? The problem there is if you ever add/remove subscriptions
and want to support rolling upgrades. If the group was subscribed to topic
A,
Jason,
The way this is handled today (in the old consumer) is by grouping several
rebalance events into one rebalance operation (similar to how we handle log
flushes using group commit). The idea is to batch several rebalance
events while one rebalance operation is in progress.
This is naturally
I think metadata churn is an interesting problem and it would be nice if we
had some approach to deal with it. I wonder if it would be sufficient to
have a setting on the client (or the server if we use centralized
assignment) which dampens the rate of rebalancing. Basically the effect
would be to
Ewen,
I agree that if there is a churn in metadata, the consumers need several
rounds of rebalances to succeed. The difference I am thinking is that with
coordinator as single source of truth, we can let the consumer finish one
round of rebalance, work for a while and start the next round of
Thanks Jiangjie, that information helps. I agree the protocol must consider
scalability. My point was that the synchronization barrier in the current
protocol already effectively limits the number of consumers since it
provides no way to gracefully handle churn. It wouldn't be worth worrying
about
Jason,
The protocol has to consider the scalability. The protocol in the wiki
means the JoinGroupResoponse size would be:
NumberOfTopics * (AvgTopicNameLength + 4) * (NumberOfConsumers)^2
To give some real number, we have 26-node Mirror Maker cluster, each with 4
consumers. That is 104 consumers
Becket,
Did you see the updates to the ConsumerEmbeddedProtocol section,
specifically on how regex subscriptions like you're describing are handled?
The updated version removes almost all the metadata in the regex case by
taking advantage of exactly the point you made in (2). The clients don't
Hey Onur and Jiangjie,
I've updated that wiki with a proposal to add regex subscriptions to the
consumer metadata. Can you have a look to see if it addresses your concern?
In general, I think we should be a little careful when we are talking about
the scalability of the protocol. Regardless of
Ewen and Jason,
Thanks for the reply. Sorry I missed the metadata hash. Yes, that is a
clever approach and would solve my concern about the data passing around. I
can see both pros and cons from doing this, though. The advantage is we
don't need the topic metadata in JoinGroupResponse anymore.
On Tue, Aug 11, 2015 at 11:29 PM, Jiangjie Qin j...@linkedin.com.invalid
wrote:
Ewen,
Thanks for the explanation.
For (1), I am more concerned about the failure case instead of normal case.
What if a consumer somehow was kick out of a group but is still consuming
and committing offsets?
On Tue, Aug 11, 2015 at 10:15 PM, Jiangjie Qin j...@linkedin.com.invalid
wrote:
Hi Jason,
Thanks for writing this up. It would be useful to generalize the group
concept. I have a few questions below.
1. In old consumer actually the partition assignment are done by consumers
themselves. We
Hey Ewen,
Onur and I discussed this a little bit more. And we are still worrying
about passing all the metadata of all consumers around.
Let's say I have a cluster has 10,000 topics, the average topic name length
is 10 bytes. In this case, the opaque metadata will have 10 * 10,000 =
100KB for
Hi Kafka Devs,
One of the nagging issues in the current design of the new consumer has
been the need to support a variety of assignment strategies. We've
encountered this in particular in the design of copycat and the processing
framework (KIP-28). From what I understand, Samza also has a number
On Tue, Aug 11, 2015 at 10:03 PM, Onur Karaman onurkaraman.apa...@gmail.com
wrote:
Just to make the conversation a bit easier (I don't think we have really
established names for these modes yet), basically with the new
KafkaConsumer today there's:
- external management, where the application
Just to make the conversation a bit easier (I don't think we have really
established names for these modes yet), basically with the new
KafkaConsumer today there's:
- external management, where the application figures out the group
management and partition assignment externally
- kafka management,
Hi Jason,
Thanks for writing this up. It would be useful to generalize the group
concept. I have a few questions below.
1. In old consumer actually the partition assignment are done by consumers
themselves. We used zookeeper to guarantee that a partition will only be
consumed by one consumer
44 matches
Mail list logo
