Kafka JMX monitoring

Hi, I want to know what the best practice to collect Kafka JMX metrics is. I haven't found a decent way to collect and parse JMX in Java (because it is too much) and I learn that there are tools like tools like jmxtrans to do this. I wonder if there is more. Thanks. Regards.

Build failed in Jenkins: kafka-2.5-jdk8 #67

See Changes: [wangguoz] HOTFIX: do not rely on file modified time in StateDirectoryTest -- [...truncated 1.55 MB...] kafka.tools.ConsoleProducerTest > testParseKeyProp START

Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

Thanks for the insight Feyman. I personally feel adding another admin client command is redundant, so picking option 1). The memberInfos struct is internal and just used for result reference purposes. I think it could still work even we overload with `removeAll` option, if that makes sense. Boyang

Build failed in Jenkins: kafka-trunk-jdk11 #1259

See Changes: [github] KAFKA-9568: enforce rebalance if client endpoint has changed (#8299) -- [...truncated 2.93 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

Re: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker

hi Cheng seems the "Next KIP Number" is not synced :( I have updated my KIP number and corrected the "Next KIP Number" thanks for the reminder. On 2020/03/19 02:32:55, "Cheng Pan" <379377...@qq.com> wrote: > Hi Chia-Ping, > > > I'm afraid the KIP number is conflict. I created (KIP-581: Value

回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

Hi, team Before going too far on the KIP update, I would like to hear your opinions about how we would change the interface of AdminClient, the two alternatives I could think of are: 1) Extend adminClient.removeMembersFromConsumerGroup to support remove all As Guochang suggested,

Build failed in Jenkins: kafka-trunk-jdk11 #1258

See Changes: [github] KAFKA-5604: Remove the redundant TODO marker on the Streams side (#8313) -- [...truncated 5.92 MB...] org.apache.kafka.streams.TopologyTestDriver

Re: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker

Hi Chia-Ping, I'm afraid the KIP number is conflict. I created (KIP-581: Value of optional null field which has default value) yesterday early, and insert it into table [KIPs under discussion]. https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+d

[jira] [Resolved] (KAFKA-9636) Simple join of two KTables fails

[ https://issues.apache.org/jira/browse/KAFKA-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-9636. - Resolution: Not A Bug Closing this ticket as "not a bug" because the test failure turned out to b

Build failed in Jenkins: kafka-trunk-jdk8 #4336

See Changes: [wangguoz] HOTFIX: do not depend on file modified time in StateDirectoryTest [github] KAFKA-9047; AdminClient group operations should respect retries and [github] KAFKA-9656; Return COORDINATOR_NOT

Build failed in Jenkins: kafka-trunk-jdk11 #1257

See Changes: [github] KAFKA-9047; AdminClient group operations should respect retries and [github] KAFKA-9656; Return COORDINATOR_NOT_AVAILABLE for older producer clients -

Re: Add a customized logo for Kafka Streams

I like the drawings too :) On Fri, Mar 13, 2020 at 3:35 AM Patrik Kleindl wrote: > Great idea, I would definitely buy the book with that on the cover :-) > best regards > Patrik > > On Fri, 13 Mar 2020 at 09:57, Becket Qin wrote: > > > I also like this one! > > > > Jiangjie (Becket) Qin > > > >

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Hello Sanjana, Thanks for the proposed KIP, I think that makes a lot of sense -- as you mentioned in the motivation, we've indeed seen many issues with regard to the frequent retries, with bounded exponential backoff in the scenario where there's a long connectivity issue we would effectively redu

[jira] [Resolved] (KAFKA-5604) All producer methods should raise `ProducerFencedException` after the first time.

[ https://issues.apache.org/jira/browse/KAFKA-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5604. -- Fix Version/s: 2.3.0 Assignee: Guozhang Wang (was: Apurva Mehta) Resolution: Fi

Re: [VOTE] KIP-574: CLI Dynamic Configuration with file input

+1 (binding). Thanks, Aneel. best, Colin On Wed, Mar 18, 2020, at 00:04, David Jacot wrote: > +1 (non-binding), thanks for the KIP! > > David > > On Mon, Mar 16, 2020 at 4:06 PM Aneel Nazareth wrote: > > > Hello all, > > > > Thanks to the folks who have given feedback. I've incorporated the

Build failed in Jenkins: kafka-trunk-jdk11 #1256

See Changes: [github] MINOR: Update Scala to 2.12.11 (#8308) [wangguoz] HOTFIX: do not depend on file modified time in StateDirectoryTest -- [...truncated 2.92 MB...]

Build failed in Jenkins: kafka-trunk-jdk8 #4335

See Changes: [github] MINOR: Update Scala to 2.12.11 (#8308) -- [...truncated 2.91 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldNotUpdateStoreForSmall

[Discuss] KIP-582 Add a "continue" option for Kafka Connect error handling

Hi all, I'd like to use this thread to discuss KIP-582 Add a "continue" option for Kafka Connect error handling, please see detail at: https://cwiki.apache.org/confluence/x/XRvcC Best, Zihan Li

[jira] [Resolved] (KAFKA-9656) TxnOffsetCommit should not return COORDINATOR_LOADING error for old request versions

[ https://issues.apache.org/jira/browse/KAFKA-9656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-9656. Fix Version/s: 2.6.0 Resolution: Fixed > TxnOffsetCommit should not return COORDINAT

[jira] [Reopened] (KAFKA-8122) Flaky Test EosIntegrationTest#shouldNotViolateEosIfOneTaskFailsWithState

[ https://issues.apache.org/jira/browse/KAFKA-8122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen reopened KAFKA-8122: Assignee: (was: Matthias J. Sax) > Flaky Test EosIntegrationTest#shouldNotViolateEosIfOneTask

Jenkins build is back to normal : kafka-trunk-jdk11 #1255

See

Build failed in Jenkins: kafka-trunk-jdk8 #4334

See Changes: [github] KAFKA-9404: Use ArrayList instead of LinkedList in Sensor (#7936) -- [...truncated 5.88 MB...] org.apache.kafka.streams.test.ConsumerRecordFactoryT

Re: [DISCUSS] KIP-577: Allow HTTP Response Headers Configured for Kafka Connect

Thanks for the proposal, Jeff. I can see how this proposal will add value. I have a few comments, most of which are asking for more detail in the KIP. 1. I do think the KIP needs to be much more explicit about what the new configuration properties will be and to use them consistently. For e

[DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker

hi this ticket is about to records partition metrics rather than topic metrics. It helps us to observe more precis metrics for specify partition. The downside is that broker needs more space to keep metrics and the origin metrics interface (tags) is broken since this ticket adds new tag "partit

[jira] [Created] (KAFKA-9733) Consider addition to Kafka's replication model

Richard Yu created KAFKA-9733: - Summary: Consider addition to Kafka's replication model Key: KAFKA-9733 URL: https://issues.apache.org/jira/browse/KAFKA-9733 Project: Kafka Issue Type: New Featur

[jira] [Created] (KAFKA-9732) Kafka Foreign-Key Joiner has unexpected default value used when a table is created via a stream+groupByKey+reduce

Adam Bellemare created KAFKA-9732: - Summary: Kafka Foreign-Key Joiner has unexpected default value used when a table is created via a stream+groupByKey+reduce Key: KAFKA-9732 URL: https://issues.apache.org/jira/br

[jira] [Resolved] (KAFKA-9404) Use ArrayList instead of LinkedList in Sensor Class

[ https://issues.apache.org/jira/browse/KAFKA-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-9404. Fix Version/s: 2.6.0 Resolution: Fixed > Use ArrayList instead of LinkedList in Sensor Class

Conditionally applying SMTs

Hi, I opened https://issues.apache.org/jira/browse/KAFKA-9673 a week or so ago sketching an idea for conditionally applying SMTs, which improve the experience for users of connectors, such as Debezium, which can send records to multiple different topics where SMTs need to be applied to for some to

Jenkins build is back to normal : kafka-2.5-jdk8 #66

See

Need for histogram type for Kafka Connect JMX Metrics

Hi, We are using kafka-connect framework in our use-case, and would also like to track metrics for connectors/task. But currently Kafka Connect JMX metrics are not supporting histogram types due to which we are not able to plot percentiles on a lot of important metrics like : kafka_connect_si

Build failed in Jenkins: kafka-trunk-jdk11 #1254

See Changes: [github] KAFKA-9625: Fix altering and describing dynamic broker configurations -- [...truncated 2.93 MB...] org.apache.kafka.streams.TopologyTestDriverTest

Regarding segment size config

Hi community, I understand that there are two configs regarding segment file size, log.segment.bytes for broker and segment.bytes for topic. The default values are both 1G and they are required to be an integer so they cannot be larger than 2G. My question is, assuming I am not making any mistakes

Build failed in Jenkins: kafka-trunk-jdk8 #4333

See Changes: [github] HOTFIX: fix flaky [github] MINOR: clean up required setup for StreamsPartitionAssignorTest (#8306) [github] KAFKA-9625: Fix altering and describing dynamic broker configurations

Build failed in Jenkins: kafka-trunk-jdk11 #1253

See Changes: [github] HOTFIX: fix flaky [github] MINOR: clean up required setup for StreamsPartitionAssignorTest (#8306) -- [...truncated 2.92 MB...] org.apache.kafka.s

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Thanks for the feedback Boyang. If there’s anyone else who has feedback regarding this KIP, would really appreciate it hearing it! Thanks, Sanjana On Tue, Mar 17, 2020 at 11:38 PM Boyang Chen wrote: > Sounds great! > > Get Outlook for iOS > __

Re: [VOTE] KIP-574: CLI Dynamic Configuration with file input

+1 (non-binding), thanks for the KIP! David On Mon, Mar 16, 2020 at 4:06 PM Aneel Nazareth wrote: > Hello all, > > Thanks to the folks who have given feedback. I've incorporated the > suggestions, and think that this is now ready for a vote: > > > https://cwiki.apache.org/confluence/display/KAF

[Discuss] KIP-581: Value of optional null field which has default value

Hi all, I'd like to use this thread to discuss KIP-581: Value of optional null field which has default value, please see detail at: https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value There are some previous discussion at: https://g