KIP-158 observations
Hi, guys! I have several thoughts about the new implemented KIP-158: "Kafka Connect should allow source connectors to set topic-specific settings for new topics" We started to use 2.6.1 version and this feature and have some observations. I have described them also in the ticket: https://issues.apache.org/jira/browse/KAFKA-10340 I cannot understand, why we even have the property "topic.creation.enable" and do not create all topics by AdminClient? Isn't it a better approach to create topics always explicitly using AdminClient than to rely on auto creation? And also I cannot see how can it has a bad impact on users. Why someone may want to set "topic.creation.enable = false"? According to KIP-158 we have an option to set replication.factor and partitions to -1 to use the broker's default value. So why we cannot always create topics by AdminClient and use these broker's default values in case connector does not have any "topic.creation" properties? And also current approach can be a problem for us - we wanted to set "auto.create.topics.enable = false" on Kafka cluster by default. But we cannot do that, because our connectors can be created with "topic.creation" parameters or without them, we cannot guarantee that. And in case connectors have not this property, we get errors. As far as I understand this will completely solve the problem described in the KAFKA-10340 ticket. And also people will be able to set "auto.create.topics.enable = false" on Kafka cluster without any impact. Please, correct me if I am wrong and give me your opinion!
Re: Request to be Assignee on JIRA
Thanks! On 2020/08/24 14:04:56, Bill Bejeck wrote: > Done. Thanks for your interest in Apache Kafka. > > -Bill > > On Mon, Aug 24, 2020 at 8:34 AM xaka...@mail.ru.INVALID > wrote: > > > Hi, guys! I'd like to provide a PR for KAFKA-10426. But I'm not in the > > Assignee list. Could you, please, add me? > > But I'm not sure that I'm writing to the right mailing list. > > My JIRA full name is Goltseva Taisiia. > > > > > > --- > > С уважением, Таисия Дорошенкова >
Re: Consumer Group rebalancing events
Hi, Srinivas Devaki ! There are Kafka JMX metrics - NumGroupsCompletingRebalance and NumGroupsPreparingRebalance. I think you can monitor them by some application and calculate statistic. As for us - we added Jolokia agent to our Kafka docker image and Jolokia expose all JMX metrics via HTTP. We run additional service which gets necessary metrics from this endpoint and analyse them. On 2020/08/23 16:39:35, "Devaki, Srinivas" wrote: > Hi Folks, > > I’m trying to find the consumer group rebalancing events, so to plot how > much time consumer groups usually take in rebalancing during application > deployments. > > I’ve tried to check the logs of `server.log`, `controller.log`, > `state-change.log`. but couldn’t find anything about rebalancing. > > > Thanks & Regards > Srinivas Devaki > sre @ zomato >
Re: Request to be Assignee on JIRA
Already added, thanks. On 2020/08/24 11:57:18, "xaka...@mail.ru.INVALID" wrote: > Hi, guys! I'd like to provide a PR for KAFKA-10426. But I'm not in the > Assignee list. Could you, please, add me? > But I'm not sure that I'm writing to the right mailing list. > My JIRA full name is Goltseva Taisiia. > > > --- > С уважением, Таисия Дорошенкова
