Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

Incidentally, whenever we do a hard compatibility break with this API, we should probably get rid of the function that iterates over every replica on the broker. That's not a scalable API for the future. We also probably should have an API to list just the directories that are moving, which is

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

Yeah. The issue with subclassing is that it's a source compatibility break, although not (I think) a binary compatibility break. I don't think it's really worth it given that it leaves us with a weird class hierarchy, and we still need to do a hard compatibility break later to fix the real pro

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi Unmesh, Yes, once the last stable offset advanced, we would consider the topic creation to be done, and then we could return success to the client. best, Colin On Thu, Jul 9, 2020, at 19:44, Unmesh Joshi wrote: > It still needs HighWaterMark / LastStableOffset to be advanced by two > records

Re: [DISCUSS] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Badai, Replying in this thread. Two possible ways come up into my head: 1. CreateTime:1592475472398|key1|3|<0>|h1=v1,h2=v2|value1 2. CreateTime:1592475472398|key1|3|offset=0|h1=v1,h2=v2|value1 I prefer the option #1 which could be accomplished by a complementary usage description. What do you t

Re: [DISCUSS] KIP-450: Sliding Windows

Leah, thanks a lot for the KIP. Very well written. The KIP does not talk about the handling of out-of-order data though. How do you propose to address this? -Matthias On 7/8/20 5:33 PM, Leah Thomas wrote: > Hi all, > I'd like to kick-off the discussion for KIP-450, adding sliding window > aggr

Build failed in Jenkins: kafka-trunk-jdk11 #1633

See Changes: [github] MINOR: Restore stream-table duality description (#8995) -- [...truncated 6.39 MB...] org.apache.kafka.streams.test.OutputVerifierTest > shouldNot

Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Hu No, the property order does not matter. And yes, anything after "<--" is my comment. Do you have any suggestions on making it more readable, while keeping it backward compatible? Can we discuss this in the DISCUSS thread? Regards Badai On Thu, Jul 9, 2020 at 8:05 PM Hu Xi wrote: > > Hi Bad

Build failed in Jenkins: kafka-trunk-jdk8 #4707

See Changes: [github] KAFKA-10249: don't try to read un-checkpointed offsets of in-memory -- [...truncated 3.18 MB...] org.apache.kafka.streams.test.ConsumerRecordFactor

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

It still needs HighWaterMark / LastStableOffset to be advanced by two records? Something like following? || <--|| HighWaterMark Response|PartitionRecord | ||

Re: [DISCUSS] 2.5.1 Bug Fix Release

Hello, all, Thanks for you patience. KAFKA-10134 was the last blocker for 2.5.1, and it has been resolved. In fact, there are currently no open bugs at all for 2.5.1. I will be proceeding to cut 2.5.1 RC0 tomorrow, and I'll send an announcement email when it is ready. Thanks, -John On Fri, Jun

Jenkins build is back to normal : kafka-trunk-jdk14 #282

See

[jira] [Created] (KAFKA-10264) Flaky Test TransactionsTest.testBumpTransactionalEpoch

Sophie Blee-Goldman created KAFKA-10264: --- Summary: Flaky Test TransactionsTest.testBumpTransactionalEpoch Key: KAFKA-10264 URL: https://issues.apache.org/jira/browse/KAFKA-10264 Project: Kafka

[jira] [Created] (KAFKA-10263) Do not create standbys for those revoking active tasks if it is not stateful

Guozhang Wang created KAFKA-10263: - Summary: Do not create standbys for those revoking active tasks if it is not stateful Key: KAFKA-10263 URL: https://issues.apache.org/jira/browse/KAFKA-10263 Projec

Build failed in Jenkins: kafka-trunk-jdk11 #1632

See Changes: [github] KAFKA-10179: Pass correct changelog topic to state serdes (#8902) [manikumar.reddy] MINOR: Create ChannelBuilder for each connection in -- [...tru

[jira] [Created] (KAFKA-10262) StateDirectory is not thread-safe

Sophie Blee-Goldman created KAFKA-10262: --- Summary: StateDirectory is not thread-safe Key: KAFKA-10262 URL: https://issues.apache.org/jira/browse/KAFKA-10262 Project: Kafka Issue Type: B

[jira] [Created] (KAFKA-10261) Introduce the KIP-478 processors with shims

John Roesler created KAFKA-10261: Summary: Introduce the KIP-478 processors with shims Key: KAFKA-10261 URL: https://issues.apache.org/jira/browse/KAFKA-10261 Project: Kafka Issue Type: Sub-t

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

On Thu, Jul 9, 2020, at 10:40, Tom Bentley wrote: > Hi Colin, > > Thanks for the detailed KIP. > > > "As described in KIP-500, the controller will store its data in the > > internal __kafka_metadata topic." > > The KIP doesn't really explain the extent to which this is a normal topic. > I know y

Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

Yuriy, thanks for the KIP update. I have one follow up thought: I checked what default Serdes we offer in the Java class `org.apache.kafka.common.serialization.Serdes` and I think it would be good if we could close the gap between the Java and Scala code and add the missing Java Serdes in Scala

[jira] [Resolved] (KAFKA-10256) Create a server gradle module for Java code needed only by servers

[ https://issues.apache.org/jira/browse/KAFKA-10256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-10256. -- Resolution: Won't Fix > Create a server gradle module for Java code needed only by servers > -

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

On Thu, Jul 9, 2020, at 04:37, Unmesh Joshi wrote: > I see that, when a new topic is created, two metadata records, a > TopicRecord (just the name and id of the topic) and a PartitionRecord (more > like LeaderAndIsr, with leader id and replica ids for the partition) are > created. > While creating

Jenkins build is back to normal : kafka-trunk-jdk8 #4705

See

[VOTE] KIP-554: Add Broker-side SCRAM Config API

Hi all, I'd like to call a vote for KIP-554: Add a broker-side SCRAM configuration API. The KIP is here: https://cwiki.apache.org/confluence/x/ihERCQ The previous discussion thread is here: https://lists.apache.org/thread.html/r69bdc65bdf58f5576944a551ff249d759073ecbf5daa441cff680ab0%40%3Cdev.k

Build failed in Jenkins: kafka-trunk-jdk14 #281

See Changes: [github] MINOR: Closing consumerGroupService resources in [github] KAFKA-10235 Fix flaky transactions_test.py (#8981) -- [...truncated 3.19 MB...] org.apac

[jira] [Created] (KAFKA-10260) Streams could recover stores in a task independently

John Roesler created KAFKA-10260: Summary: Streams could recover stores in a task independently Key: KAFKA-10260 URL: https://issues.apache.org/jira/browse/KAFKA-10260 Project: Kafka Issue Ty

[jira] [Created] (KAFKA-10259) KIP-554: Add Broker-side SCRAM Config API

Ron Dagostino created KAFKA-10259: - Summary: KIP-554: Add Broker-side SCRAM Config API Key: KAFKA-10259 URL: https://issues.apache.org/jira/browse/KAFKA-10259 Project: Kafka Issue Type: New F

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi Colin, Thanks for the detailed KIP. >> "As described in KIP-500, the controller will store its data in the internal __kafka_metadata topic." The KIP doesn't really explain the extent to which this is a normal topic. I know you say in the Networking section that the only time a client should c

Jenkins build is back to normal : kafka-trunk-jdk14 #280

See

Build failed in Jenkins: kafka-trunk-jdk11 #1630

See Changes: [manikumar.reddy] KAFKA-10109: Fix double AdminClient creation in AclCommand [github] MINOR: Define the term tombstone, since it's used elsewhere in the docs -

[jira] [Created] (KAFKA-10258) Get rid of use_zk_connection flag in kafka.py public methods

Vinoth Chandar created KAFKA-10258: -- Summary: Get rid of use_zk_connection flag in kafka.py public methods Key: KAFKA-10258 URL: https://issues.apache.org/jira/browse/KAFKA-10258 Project: Kafka

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi Jun, Thanks for the replies and feedback on design and giving input. We are coming close to finish the implementation. We also did several perf tests as well at our peak production loads and with tiered storage we didn't see any degradation on write throughputs and latencies. Ying alre

[jira] [Resolved] (KAFKA-10235) Fix flaky transactions_test.py

[ https://issues.apache.org/jira/browse/KAFKA-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-10235. - Fix Version/s: 3.0.0 Resolution: Fixed merged the PR to trunk. > Fix flaky transactions_test.py

Build failed in Jenkins: kafka-trunk-jdk8 #4704

See Changes: [manikumar.reddy] KAFKA-10109: Fix double AdminClient creation in AclCommand [github] MINOR: Define the term tombstone, since it's used elsewhere in the docs --

[jira] [Created] (KAFKA-10257) system test kafkatest.tests.core.security_rolling_upgrade_test fails

Jun Rao created KAFKA-10257: --- Summary: system test kafkatest.tests.core.security_rolling_upgrade_test fails Key: KAFKA-10257 URL: https://issues.apache.org/jira/browse/KAFKA-10257 Project: Kafka I

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

Hi Dongjin, Thanks for updating the link and sorry that you put effort into writing your own KIP for this. I was aware of KAFKA-8794, but since the PR was about javadocing the internal classes I felt it wasn't quite the same thing. All the same, I should have pinged you on KAFKA-8794, sorry. I di

[jira] [Created] (KAFKA-10256) Create a server gradle module for Java code needed only by servers

Colin McCabe created KAFKA-10256: Summary: Create a server gradle module for Java code needed only by servers Key: KAFKA-10256 URL: https://issues.apache.org/jira/browse/KAFKA-10256 Project: Kafka

Re: [DISCUSS] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

Hi Colin, Thanks for the quick response. Sure, the problem is that the internal classes (DescribeLogDirsResponse.[LogDirInfo, ReplicaInfo], or 'the old ones') are exposed to the public API, not the class itself. You are right. However, deprecating DescribeLogDirsResult#[all, values] and definin

Re: [VOTE] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

Thanks Yuriy, I'm +1 (binding) -John On Wed, Jul 8, 2020, at 23:08, Yuriy Badalyantc wrote: > Hi everybody > > I would like to start a vote for KIP-616: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-616%3A+Rename+implicit+Serdes+instances+in+kafka-streams-scala > > This KIP fixes na

[jira] [Created] (KAFKA-10255) Fix flaky testOneWayReplicationWithAutorOffsetSync1 test

Luke Chen created KAFKA-10255: - Summary: Fix flaky testOneWayReplicationWithAutorOffsetSync1 test Key: KAFKA-10255 URL: https://issues.apache.org/jira/browse/KAFKA-10255 Project: Kafka Issue Type

Build failed in Jenkins: kafka-trunk-jdk14 #279

See Changes: [manikumar.reddy] KAFKA-10220: Add null check for configurationKey in [manikumar.reddy] KAFKA-10109: Fix double AdminClient creation in AclCommand -- [...tr

Jenkins build is back to normal : kafka-trunk-jdk8 #4703

See

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

I see that, when a new topic is created, two metadata records, a TopicRecord (just the name and id of the topic) and a PartitionRecord (more like LeaderAndIsr, with leader id and replica ids for the partition) are created. While creating the topic, log entries for both the records need to be commit

[jira] [Created] (KAFKA-10254) 100% cpu usage by kafkaConsumer poll , when broker can‘t be connect

xiaotong.wang created KAFKA-10254: - Summary: 100% cpu usage by kafkaConsumer poll , when broker can‘t be connect Key: KAFKA-10254 URL: https://issues.apache.org/jira/browse/KAFKA-10254 Project: Kafka

回复: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Hi Badai, Thanks for the KIP. A quick question from me: "CreateTime:1592475472398|key1|3|0|h1=v1,h2=v2|value1<-- offset 3, partition 0" Seems the partition follows the offset. My question is, does the property order matter? The partition is always printed following the offset no matter the

Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

David Thanks for the vote. That is a good idea. Will start another KIP once this one is done. Regards Badai On Thu, Jul 9, 2020 at 6:46 PM David Jacot wrote: > > Hi Badai, > > Thanks for the KIP. I think that it is a nice improvement so I am +1 > (non-binding). > > Long term, I wonder if we cou

[jira] [Resolved] (KAFKA-10109) kafka-acls.sh/AclCommand opens multiple AdminClients

[ https://issues.apache.org/jira/browse/KAFKA-10109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-10109. --- Fix Version/s: 2.7.0 Resolution: Fixed Issue resolved by pull request 8808 [https://github.co

Re: [VOTE] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

+1 (binding) Thanks On Thu, Jul 9, 2020 at 9:11 AM David Jacot wrote: > > I couldn't think of a better alternative either. Thanks for the KIP! > > +1 (non-binding) > > On Wed, Jul 8, 2020 at 12:42 PM Manikumar wrote: > > > +1 (binding) > > > > Thanks for the KIP. > > > > On Tue, Jul 7, 2020 at 8

[jira] [Created] (KAFKA-10253) Kafka Connect gets into a rebalance loop

Konstantin Lalafaryan created KAFKA-10253: - Summary: Kafka Connect gets into a rebalance loop Key: KAFKA-10253 URL: https://issues.apache.org/jira/browse/KAFKA-10253 Project: Kafka Is

Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Hi Badai, Thanks for the KIP. I think that it is a nice improvement so I am +1 (non-binding). Long term, I wonder if we could adopt a formatting system similar to kafkacat. It would reduce the number of properties that one has to set and also allow more powerful formatting. That could be done as

Re: [VOTE] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

I couldn't think of a better alternative either. Thanks for the KIP! +1 (non-binding) On Wed, Jul 8, 2020 at 12:42 PM Manikumar wrote: > +1 (binding) > > Thanks for the KIP. > > On Tue, Jul 7, 2020 at 8:19 PM Colin McCabe wrote: > > > Thanks, Tom. > > > > I tried to think of a better way to do

[jira] [Created] (KAFKA-10252) MeteredTimestampedKeyValueStore not setting up correct topic name for GlobalKTable associating StateStores

Gordon Wang created KAFKA-10252: --- Summary: MeteredTimestampedKeyValueStore not setting up correct topic name for GlobalKTable associating StateStores Key: KAFKA-10252 URL: https://issues.apache.org/jira/browse/KAFKA